Adjustable variable resolution inspection systems and methods

ABSTRACT

Camera heads configured to provide digitally articulated images or video, at adjustable resolutions and/or offsets and orientations, to a camera control unit (CCU) or other electronic computing system for display, storage, and/or transmission to other systems are disclosed.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims priority to co-pendingU.S. patent application Ser. No. 13/754,767, entitled ADJUSTABLEVARIABLE RESOLUTION INSPECTION SYSTEMS AND METHODS, filed on Jan. 30,2013, which claims priority under 35 U.S.C. §119(e) to U.S. ProvisionalPatent Application Ser. No. 61/592,524, entitled ADJUSTABLE VARIABLERESOLUTION INSPECTION SYSTEMS AND METHODS, filed on Jan. 30, 2012. Thecontent of these applications is incorporated by reference herein intheir entirety for all purposes.

FIELD

This disclosure relates generally to devices, systems, & methods forinspecting pipes or other cavities using images and video, which may becombined with sensor data, audio signals, location, orientation, and/orpositioning information, and/or other data or information. Morespecifically, but not exclusively, the disclosure relates to a camerahead with one or more electronic imaging elements, including electronicimage sensors and associated optics, configured to provideelectronically generated images or video, at adjustable resolutionsand/or offsets, translations, zooms, and/or orientations, to a cameracontrol unit (CCU) or other electronic computing system or device fordisplay, storage, and/or re-transmission.

BACKGROUND

Pipe inspection systems for examining the interior of pipes, conduits,and other cavities or voids are known in the art. In order to correctlydiagnose a defect within the interior of pipes, conduits, and othervoids, a video camera linked to a push cable is generally employed. In atypical configuration, a rugged camera head connected to a push cable issent through a pipe, and the camera head transmits video signals in ananalog signaling format and at a fixed resolution along a transmissionmedium to view the scene on a remote monitoring system. These systemsgenerally do not provide the capability of generating and sendingvariable resolution images, tiled images, zoomed or stitched images, HDRimages, or other adjusted images to a camera control unit (CCU) or otherelectronic computing system or display device, nor do they providevariable orientation and/or resolution images based on conditionsassociated with the camera head or based on user inputs.

Accordingly, there is a need in the art to address the above-describedas well as other problems.

SUMMARY

The present disclosure relates generally to pipe inspection systems,apparatus, and methods that may be used for imaging of pipes, conduits,and/or other cavities or spaces using zooming, tiling, translations,rotations, high dynamic range (HDR) image processing, and other imagingoperations. Components of a self-leveling pipe inspection system mayinclude camera heads, output signal transmission media, camera controlunits (CCUs), displays, audio transducers, position, orientation,location, and/or others sensors, as well as related components. Althoughthe camera heads, CCUs, and other devices disclosure herein arepresented in the context of a pipe inspection system, the teachingsherein can similarly be applied to other camera and imaging applicationsin the art.

In one aspect, the present disclosure relates to a method of providing avariable-resolution visual display, such as in an inspection system suchas a buried pipe or conduit visual inspection system. The method mayinclude, for example, capturing a first image, covering a first field ofview, in an electronic imaging element, including an image sensor andassociated optics, of a camera head. The camera head may include oneimaging element or a plurality of imaging elements. In embodiments witha plurality of imaging elements, fields of view of ones of the imagingelements may overlap, and/or the optical axes of the imaging elementsmay be non-parallel, such as by being divergent. The method may includeconverting the first image to a first analog signal. The method mayfurther include providing the first analog signal to an electroniccomputing system, such as a camera control unit (CCU), portablecomputer, tablet, or other device. The method may include sensing acondition associated with the camera head. The method may furtherinclude generating, responsive to the sensing, ones of a plurality oftiled images corresponding to tiled subsets of the first field of view.The method may include converting the plurality of tiled images to asecond analog signal. The method may further include providing thesecond analog signal to the CCU. In embodiments with multiple imagingelements, the first image and tiled images may be generated based onpixel data provided from multiple image sensors.

In another aspect, the disclosure relates to apparatus & systems forimplementing the above-described method or other methods disclosedherein, in whole or in part.

In another aspect, the disclosure relates to means for implementing theabove-described method or other methods described herein, in whole or inpart.

In another aspect, the disclosure relates to computer readable mediaincluding instructions for causing a processor to implement theabove-described method or other methods described herein, in whole or inpart.

Various additional features, aspects, and embodiment and implementationdetails are further described below in conjunction with the appendedDrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure may be more fully appreciated in connection withthe following detailed description taken in conjunction with theaccompanying drawings, wherein:

FIG. 1A is a block diagram illustrating details of an embodiment of apipe inspection system;

FIG. 1B is a block diagram illustrating details of an embodiment ofimage processing and control flow in a camera head of a pipe inspectionsystem.

FIG. 1C is a block diagram illustrating details of multiple data setsfrom multiple image sensors which may be processed together in anembodiment of a single camera head.

FIG. 2 illustrates details of an embodiment of a signal processingmodule for use in a camera head of a pipe inspection system such asshown in FIG. 1A;

FIG. 3 illustrates details of an example image reorientation processconsistent with aspects of the present invention;

FIG. 4 illustrates details of an embodiment of a process for providingorientation adjusted images or video in a pipe inspection system such asshown in FIG. 1A;

FIG. 5 illustrates details of one embodiment of signal processing togenerate or extract an orientation adjusted image from a memory;

FIG. 6 illustrates details of an image sensor configuration that may beused to generate tiled image or video data;

FIGS. 7 & 8 illustrate details of the image sensor of FIG. 6 and acorresponding scanned analog video frame;

FIG. 9A illustrates details of example tiling of the pixels of the imagesensor of FIG. 6;

FIG. 9B illustrates details of a multi sensor array sub-divided intonine tiles;

FIG. 9C illustrates details of an exemplary 4X zoom/pan video using oneor more imagers;

FIG. 9D illustrates details of how imagers with overlapping FOV may havedifferent exposures and may be combined to produce a single High DynamicRange (HDR) image or video stream;

FIG. 9E is a block diagram illustrating details of an embodiment ofdigital articulation of a multiple image sensor data combined togenerate a single image or string of images of greater field of view ordetail than can be provided by a single imager.

FIG. 9F is a block diagram illustrating details of an embodiment ofstandardized analog video being used as a general purpose digitaltransmission scheme.

FIG. 9G illustrates details of an embodiment of a process for providingoutput image data across image sensor boundaries in an image sensorarray.

FIG. 10 illustrates details of an embodiment of tiled video processing;

FIG. 11 illustrates details of an embodiment of a high-resolution stillmode of tiled video processing;

FIG. 12 illustrates details of an embodiment of anintermediate-resolution still mode of tiled video processing;

FIG. 13 illustrates details of an embodiment of a high resolution lowframe rate video mode of tiled video processing;

FIG. 14 illustrates details of an embodiment of tile generation andprocessing to provide high resolution video or images;

FIG. 15 illustrates details of an embodiment of tile generation andprocessing to provide a selected tile or tiles;

FIG. 16 illustrates details of an embodiment of a process for providinga variable resolution display in an inspection system;

FIG. 17 illustrates details of an embodiment of a process for providinga selectable tiled display in an inspection system;

FIG. 18 illustrates details of an embodiment of a process for providinga CCU automatically generated variable resolution display in aninspection system;

FIG. 19 illustrates details of a camera head embodiment to providedigital articulation;

FIG. 20 illustrates details of the camera head embodiment of FIG. 19,taken from the front;

FIG. 21 is a section view of the camera head embodiment of FIG. 20,taken from line 21-21 (through imagers);

FIG. 22 is a section view of the camera head embodiment of FIG. 20,taken from line 22-22 (through lasers);

FIG. 23 is an exploded view of the camera head embodiment of FIG. 20;

FIG. 24 is an exploded view of a front bezel assembly as illustrated inFIGS. 19-23;

FIG. 25 is an exploded view of an imager PCB assembly as illustrated inFIGS. 21-23;

FIG. 26 illustrates how multiple lasers originating at the camera headare used to create markers on a pipe for detecting shape, orientation,or other image processing reference information;

FIG. 27 illustrates how the laser dot location can be processed andcorrelated between the multiple imagers to the dimensions and/or shapeof the inside of a pipe; and

FIG. 28 illustrates how certain areas within the entire FOV of thecomposite (multiple integrated imagers) camera will have overlap betweenimagers, and how some areas within the entire FOV will only be visibleto a sub-set of the imagers.

FIG. 29 illustrates an example image sensor as may be used in imagesensor array embodiments.

FIG. 30 illustrates an image sensor array embodiment with overlappingfields of view.

FIGS. 31A-31H illustrate example viewport or tile transitions acrossimage sensor boundaries in the image sensor array of FIG. 30.

DETAILED DESCRIPTION OF EMBODIMENTS Overview

The present disclosure relates generally to pipe inspection systems,apparatus, and methods that may be used for imaging of pipes, conduits,and/or other cavities or spaces using zooming, tiling, translations,rotations, high dynamic range (HDR) image processing, and stitching ofimages from a single image sensor or multiple image sensors, and/orother imaging operations based on images generated in one or more imagesensors. Components of a self-leveling pipe inspection system mayinclude camera heads, output signal transmission media, camera controlunits (CCUs) or other coupled electronic computing systems, displays,audio transducers, position, orientation, location, and/or otherssensors, as well as related components.

In an exemplary embodiment, a pipe inspection system camera head mayinclude elements or components for providing a digital image, such as animaging element or elements including an electronic image sensor orsensor array using CCD, CMOS, or other image or video sensing andcapture technologies and associated optics, along with an orientationand/or position sensing module for providing orientation and/or positiondata associated with the position and/or orientation of the imagesensor, which may be relative to a reference orientation such as thatdue to gravity (e.g., up/down relative to the earth's surface or anormal upright orientation). Output images or video from the camera heador CCU may be based on images captured by multiple image sensors, suchas a video stream including images captured from a first image sensorbefore a transition of a viewport through an overlap area, and anotherimage sensor during or subsequent to the transition. The images may beregistered between the image sensors using light markers or targets,such as laser dots or grating generated targets projected in an areabeing viewed by the camera head.

Output images and/or video data from the image sensor may be provided,along with the orientation/position data and/or other data orinformation, such as audio information, other sensor data, and the like,to an image processing module or modules including one or moreprocessing elements, which may then provide an output video signal thatis adjusted in the camera head for a desired orientation, such as anupright orientation relative to gravity.

The output video signal may then be provided to a camera control unit(CCU) or other system or device that includes display and/or storagecapability. A visual display corresponding to the output video signalmay then be provided, such as by rendering the output video signal on adisplay device of the CCU or other system such as an LCD display, and/oron other devices such as printer, and/or on another video or imageoutput device. In addition, the image or video may also be stored forfuture display, output, transmission, processing, and/or analysis.Operator input at the CCU or other control device may be provided to thecamera head to control digital articulation (i.e., translations, zooms,rotations, etc. of the output viewport and corresponding images or videogenerated at the camera head) and/or the camera head may be articulatedautomatically, such as in response to sensor input from a position,orientation, or location sensor, such as an accelerometer, compasssensor, gravitations sensor, location or position sensor, or othersensors in or on the camera head.

For example, in one aspect, the present disclosure relates to a methodof providing a variable-resolution visual display, such as in aninspection system such as a buried pipe or conduit visual inspectionsystem. The method may include, for example, capturing a first imagecovering a first field of view in an electronic imaging element,including an image sensor and associated optics of a camera head. Thecamera head may include one imaging element or a plurality of imagingelements. In embodiments with a plurality of imaging elements, fields ofview of ones of the imaging elements may overlap, and/or the opticalaxes of the imaging elements may be non-parallel, such as by beingoutward divergent. The method may include converting the first image toa first analog signal. The method may further include providing thefirst analog signal to an electronic computing system, such as a cameracontrol unit (CCU), portable computer, tablet, or other device. Themethod may include sensing a condition associated with the camera head.The method may further include generating, responsive to the sensing,ones of a plurality of tiled images corresponding to tiled subsets ofthe first field of view. The method may include converting the pluralityof tiled images to a second analog signal. The method may furtherinclude providing the second analog signal to the CCU.

The method may include, for example, receiving the first analog signal,such as at the CCU or other communicatively coupled electronic computingsystem. The method may further include providing a first visual outputcorresponding to the first analog signal on a display device, such as onan LCD or other display element integral with or coupled to the CCU orother device, at a first resolution. The method may further includereceiving the second analog signal, such as at the CCU. The method mayfurther include combining the plurality of tiled images of the secondanalog signal to provide a second visual output on the display device ata second resolution higher than the first resolution.

The method may include, for example, generating a second plurality oftiled images corresponding to tiled subsets of the first field of view.The method may further include converting the second plurality of tiledimages to a third analog signal. The method may further includeproviding the third analog signal to the CCU or other device. The methodmay include combining the second plurality of tiled images of the thirdanalog signal to provide a third visual output on the display device.The third visual output may be provided at a third resolution, which maybe different than the first and the second resolutions.

The second visual output may, for example, be provided as a zoomed-infield of view relative to the first visual output. The first image andthe second image may be provided as a sequence of images and/or asframes of a video signal. The first image and the aggregate of the tiledsubsets may correspond to the same field of view. The aggregate of thetiled subsets may correspond to a narrower field of view than the firstimage. The aggregate of the tiled subsets may correspond to a widerfield of view than the first image. The aggregate of the tiled subsetsmay correspond to a rotation and/or translation of the field of view ofa previous image or images. The first image and the aggregate of thetiled subsets may, for example, fully or approximately correspond to thefield covered by the full frame of the image sensor.

The plurality of tiled images may, for example, include four tiledimages. Each of the four tiled images may correspond to approximatelyone quarter of the first field of view. The plurality of tiled imagesmay include nine tiled images. Each of the nine tiled images maycorrespond to approximately one ninth of the first field of view.

The condition may, for example, correspond to a motion of the camerahead. The motion may be a stop motion. The motion may be a speedthreshold. The motion may be a start motion. The condition may be arotation of the image sensor relative to a reference orientation, whichmay be provided by a sensor such as an accelerometer, a compass sensor,a gravitational sensor, a radio-positioning or GPS sensor, or othermotion, position, or location sensors. The condition may relate to amotion of the camera head. The motion of the camera head may be sensedusing an accelerometer, gyroscopic sensor, or compass sensor. The motionof the camera head may be sensed using a cable counter sensor or othercable deployment or retraction sensor. The reference orientation may bean up-down gravitational orientation. The reference orientation may be aheight or distance orientation within a pipe or conduit.

The method may, for example, include providing data associated with theplurality of tiled images. The data may relate to a position and/or sizeof the ones of the plurality of tiled images. The data associated withthe plurality of tiles may be other sensor data captured in conjunctionwith the tiled images. The ones of the plurality of tiled images may beat a lower resolution than the first image. The first image and the onesof the plurality of tiled images may be provided fully or partially as adigital signal rather than an analog signal. Audio data or informationmay be provided in conjunction with the plurality of tiled images. Thefirst image and the ones of a plurality of tiled images may be stored ona digital storage medium. The first image and the ones of a plurality oftiled images may be provided as a hard copy output. The data may relateto the transmission mode currently being used to transmit the tiledimages. The transmission mode may be one of a single frame mode or atiled mode. The data may relate to one or more sensing conditionsassociated with the image being transmitted. The data may relate toimager status information at the time of image capture. The data mayrelate to camera head status information at the time of image capture. Asubset of the plurality of tiled images may be used to generate thezoomed-in field of view of the second visual output.

The method may include, for example, capturing a subsequent image in asecond electronic imaging element of the camera head, and generating athird analog signal. The third analog signal may be based at least inpart on the subsequent image. The third analog signal may be basedentirely on the subsequent image. The third analog signal may begenerated based on a combination of the subsequent image captured by thesecond image sensor and one or more pixels of a previous image capturedby the image sensor. The subsequent image may be captured during orsubsequent to a transition of a viewport or tile across an overlap areabetween the first image sensor and the second image sensor.

The method may include, for example, generating an output image sequenceor video stream from two or more imaging elements in the camera head.The output image sequence or video stream may be switched from pixeldata of a first imaging element of the two or more imaging elements topixel data from a second imaging element of the plurality of imagingelements, such as during transition of a viewport through an overlapregion of the field of view or coverage area of the imaging elements.

The method may include, for example, projecting, from the camera head, alight marker on an area being inspected. The light marker may be a laserdot or plurality of laser dots. The light marker may be a target graphicgenerated by a grating or other optical element. The target, such alaser dot or plurality of laser dots or other graphic, may be strobed insynchronization with a video frame rate associated with the imagingelements or a processing element of the camera head. The strobing mayprovide the target during only certain image captures or video frames.The method may further include registering images from a first imagesensor of a plurality of image sensors with one or more other imagesensors of the plurality of image sensors using the projected target.The registering images may be done with images generated duringtransition of the viewport through the overlap region.

The method may include, for example, capturing a subsequent plurality ofimages in the camera head at different exposures and generating, basedon the subsequent plurality of images, a high dynamic range (HDR) image.The method may further include generating a plurality of HDR images andgenerating a video stream based on the plurality of HDR images. Themethod may further include projecting, from the camera head, a lightmarker or target on an area being inspected. The method may furtherinclude generating the HDR image based in part on registration of theimages at different exposures using the projected light marker ortarget.

The method may include, for example, capturing a subsequent image in asecond electronic image sensor of the camera head and generating astereoscopic image pair or stereoscopic video stream based on thesubsequent image and an additional image captured by the image sensor.The method may further include projecting, from the camera head, a lightmarker or target on an area being inspected. The method may furtherinclude generating the stereo pair based in part on registration of thesubsequent image and additional image using the projected light markeror target.

The method may include, for example, capturing a subsequent image in asecond electronic image sensor of the camera head and generating astitched composite image or video stream based on the subsequent imageand an additional image captured by the image sensor. The method mayfurther include projecting, from the camera head, a light marker ortarget on an area being inspected. The method may further includegenerating the stitched composite image based in part on registration ofthe subsequent image and additional image using the projected lightmarker or target.

In another aspect, the disclosure relates to a camera head. The camerahead may include, for example, a body, a sensor module configured tosense a condition associated with the body, and an electronic imagingelement, including an image sensor and associated optics, disposed inthe body. The image sensor may be configured to capture a first imagecovering a first field of view, and capture, responsive to the sensing,data to be used to derive ones of a plurality of tiled imagescorresponding to tiled subsets of the first field of view. The camerahead may further include an electronics module. The electronics modulemay be configured to convert the first image to a first analog signal.The electronics module may be further configured to provide the firstanalog signal to a camera control unit (CCU). The electronics module maybe configured to extract the plurality of tiled images. The electronicsmodule may be further configured to convert the plurality of tiledimages to a second analog signal. The electronics module may be furtherconfigured to provide the second analog signal to the CCU.

The camera head may, for example, include a second electronic imagingelement disposed in the body. The imaging elements may be oriented suchthat a field of view of the imaging element may overlap with a field ofview of the second imaging element. The optical axis of the firstimaging element and the second imaging element may be disposed in anon-parallel orientation. The optical axes of the first imaging elementand the second imaging element may be divergent.

The camera head may, for example, include a plurality of additionalimaging elements. The imaging element may be oriented along a camerahead centerline, and additional imaging elements may be oriented withoutward divergent optical axes relative to the imaging element. Theoptical axes of ones of the plurality of additional imaging elements maybe non-parallel. The optical axes of ones of the plurality of additionalimaging elements may be divergent.

The camera head may, for example, include a lighting element to projecta light marker on an area being viewed by the camera head. The lightingelement may be a laser or other light emitting device. A grating orother optical element may be used to generate the target.

In another aspect, the disclosure relates to a camera head. The camerahead may include, for example, a body and a plurality of electronicimaging elements, each including an image sensor and associated optics,disposed in the body. The camera head may further include a sensormodule configured to sense a condition associated with the body. Thecamera head may further include an electronics module. The electronicsmodule may include one or more processing elements to convert imagesfrom the image sensors of the plurality of imaging elements to outputimage sequences or video streams. The electronics module may be furtherconfigured to generate analog or digital video output based on the imagesequences or video streams. The analog or digital video output may beprovided to a CCU or other electronic computing system for display,storage, or transmission to other systems.

Ones of the plurality of imaging elements may, for example, be orientedsuch that a field of view of the imaging element overlaps with a fieldof view of others of the imaging elements. The optical axes of theimaging elements may be disposed in a non-parallel orientation. Theoptical axes of the imaging elements may be divergent. A first of theplurality of imaging elements may be oriented along a camera headcenterline, and additional imaging elements may be oriented with outwarddivergent optical axes relative to the first imaging element. The camerahead may further including lighting elements to project a light markeror target on an area being viewed.

The camera head may, for example, include a second electronic imagingelement disposed in the body. The imaging elements may be oriented suchthat a field of view of the first imaging element may overlap with afield of view of the second imaging element. The optical axis of thefirst imaging element and the second imaging element may be disposed ina non-parallel orientation. The optical axes of the first imagingelement and the second imaging element may be divergent.

The camera head may, for example, include a plurality of additionalimaging elements. The imaging element may be oriented along a camerahead centerline, and additional imaging elements may be oriented withoutward divergent optical axes relative to the imaging element. Theoptical axes of ones of the plurality of additional imaging elements maybe non-parallel. The optical axes of ones of the plurality of additionalimaging elements may be divergent.

The camera head may, for example, include a lighting element to projecta light marker or target on an area being viewed by the camera head. Thelighting element may be a laser or other light emitting device. Agrating or other optical element may be used to generate the marker ortarget.

In another aspect, the disclosure relates to a Camera Control Unit (CCU)for controlling operation of a camera head, such as positioning,rotation, translation, deployment, withdrawal, zooming, panning,lighting from the camera head, and/or other control functions associatedwith operation of the camera head and/or inspection system. The CCU mayinclude, for example, a body, an electronics module to receive andprocess analog and/or digital signals including image and/or videoand/or audio data provided from the camera head, and an electronicsmodule configured to extract the image or video data from the analogsignals or digital signals. The CCU may further include a processingmodule configured to interpret the analog signal being received, or atranslated version of the analog signal, to format data containedtherein for display, storage, and/or transmission, such as storage ofthe data on a memory of the CCU and/or display images, video, audio,and/or sensor data on the display or audio output device of the CCU,and/or to send the images, video, audio, and/or sensor data from the CCUto another electronic computing system. The processing module may befurther configured to receive data along with or in conjunction with thevideo signal and use the data to display, store, and/or render a visualdisplay associated with the images or video.

In another aspect, the disclosure relates to a pipe or conduitinspection system. The system may include, for example, a camera controlunit (CCU), a camera head including a camera head body, a sensor moduleconfigured to sense a condition associated with the body, and anelectronic imaging element, including an image sensor and associatedoptics, disposed in the body. The image sensor may be configured tocapture a first image covering a first field of view, and capture,responsive to the sensing, data to be used to derive ones of a pluralityof tiled images corresponding to tiled subsets of the first field ofview. The system may further include an electronics module. Theelectronics module may be configured to convert the first image to afirst analog signal, provide the first analog signal to a camera controlunit (CCU), extract the plurality of tiled images, convert the pluralityof tiled images to a second analog signal, and provide the second analogsignal to the CCU. The system may further include a transmission mediato carry the analog signals from the camera head to the CCU. The CCU mayinclude a CCU body, an electronics module configured to receive andprocess analog signals including image or video data provided from thecamera head, an electronics module configured to extract the image orvideo data from the analog signals, and a CCU processing moduleconfigured to interpret the analog signal being received, or atranslated version of the analog signal, and format data containedtherein for display, storage, and/or transmission. The camera head mayfurther include a plurality of imaging elements and/or a lightingelement for generating markers on an area being viewed.

In another aspect, the disclosure relates to a camera head. The camerahead may include, for example, means for capturing a first image,covering a first field of view, in an electronic image sensor, means forconverting the first image to a first analog signal, means for providingthe first analog signal to a camera control unit (CCU), means forsensing a condition associated with the image sensor, means forgenerating, responsive to the sensing, ones of a plurality of tiledimages corresponding to tiled subsets of the first field of view, meansfor converting the plurality of tiled images to a second analog signal,and means for providing the second analog signal to the CCU. The camerahead may further include additional imaging means and/or means forprojecting a light target on an area being viewed.

In another aspect, the disclosure relates to a processor readable mediumincluding instructions for causing a processor or computer to receive afirst image covering a first field of view from an electronic imagesensor, convert the first image to a first analog signal, provide thefirst analog signal to a camera control unit (CCU), receive a sensorsignal associated with a sensed condition related to the image sensor,generate, responsive to the sensing, ones of a plurality of tiled imagescorresponding to tiled subsets of the first field of view, convert theplurality of tiled images to a second analog signal, and provide thesecond analog signal to the CCU.

In another aspect, the disclosure relates to a method of providing avisual display in an inspection system. The method may include, forexample, providing a first image from an electronic image sensor of acamera head, at a first resolution, to a camera control unit (CCU),sensing a condition associated with the image sensor, generating,responsive to the sensing, a second image from the image sensor at asecond resolution different from the first resolution, and providing thesecond image to the CCU.

The method may, for example, further include providing a first visualoutput corresponding to the first image on a display device, andproviding a second visual output corresponding to the second image onthe display device. The first image and the second image may be providedas frames of a video signal. The second resolution may be lower than thefirst resolution. The first image may correspond fully or approximatelywith the field covered by the full frame of the image sensor, and thesecond image may correspond with a subset of the field covered by thefull frame. The subset may be a tile of the full frame. The tile may beapproximately one quarter of the full frame. The tile may beapproximately one ninth of the full frame. The tile may be approximatelycentered within the field of view of the image sensor.

The method may further include, for example, receiving a tile selectionsignal from the CCU. The tile position may be based at least in part onthe tile selection signal. The tile selection signal may include azoom-in signal. The tile selection signal may include a zoom-out signal.The tile selection signal may include a translation signal, such as asignal to translate the provided images or video in an X and/or Ydirection on the display screen (i.e., where the Z-direction is in andout of the screen). The tile selection signal may include a rotationsignal, such as a signal to rotate the orientation of the display aboutthe center of the tile or another center point within the inspectionarea being imaged by the camera head.

The method may further include, for example, receiving a tile selectionsignal from the CCU, such as at the camera head. The tile size and/ortile position may be based at least in part on the tile selectionsignal. The condition may correspond to a motion of the image sensor.The motion may be a stop motion. The motion may be a speed threshold.The motion may be a start motion. The condition may be a rotation of theimage sensor relative to a reference orientation. The referenceorientation may be an up-down gravitational orientation.

The first image may, for example, correspond with approximately the fullframe of the image sensor and the second image may correspond with asubset of the full frame. The subset of the full frame may be selectedbased at least in part on the motion of the image sensor. The method mayfurther include providing data associated with the second image to theCCU. The second image may correspond to a tiled subset of the full frameand the data may correspond to a position and/or size of the tiledsubset.

In another aspect, the disclosure relates to a camera head. The camerahead may include, for example, a body, a sensor module configured tosense a condition associated with the body, and an electronic imagingelement, including an image sensor and associated optics, disposed inthe body. The imaging element may be configured to capture a first imagecovering a first field of view and capture, responsive to the sensing, asecond image. The camera head may further include an electronics moduleconfigured to convert the first image to a first analog signal, providethe first analog signal to a camera control unit (CCU), and convert thesecond image to a second analog signal. The second analog signal mayhave a second resolution different from the first resolution. The secondanalog signal may be provided to the CCU.

The camera head may, for example, include a second electronic imagingelement disposed in the body. The imaging elements may be oriented suchthat a field of view of the first imaging element may overlap with afield of view of the second imaging element. The optical axis of thefirst imaging element and the second imaging element may be disposed ina non-parallel orientation. The optical axes of the first imagingelement and the second imaging element may be divergent.

The camera head may, for example, include a plurality of additionalimaging elements. The imaging element may be oriented along a camerahead centerline, and additional imaging elements may be oriented withoutward divergent optical axes relative to the imaging element. Theoptical axes of ones of the plurality of additional imaging elements maybe non-parallel. The optical axes of ones of the plurality of additionalimaging element may be divergent.

The camera head may, for example, include a lighting element to projecta light marker on an area being viewed by the camera head. The lightingelement may be a laser or other light emitting device. A grating orother optical element may be used to generate the target.

In another aspect, the disclosure relates to a Camera Control Unit(CCU). The CCU may include, for example, a body, an electronics moduleto receive and process analog signals including image or video dataprovided from a camera head, an electronics module configured to extractthe image or video data from the analog signals, and a processing moduleconfigured to interpret the analog signal being received, or atranslated version of the analog signal, and format data containedtherein for display, storage, and/or transmission. The processing modulemay be further configured to receive data in conjunction with the videosignal and use the data to either display, store, or produce a visualdisplay associated with the images.

In another aspect, the disclosure relates to a pipe inspection system.The pipe inspection system may include, for example, a camera controlunit and a camera head. The camera head may include a body, a sensormodule configured to sense a condition associated with the body, and anelectronic image sensor disposed in the body. The image sensor may beconfigured to capture a first image covering a first field of view, andcapture, responsive to the sensing, a second image. The system mayfurther include an electronics module. The electronics module may beconfigured to convert the first image to a first analog signal, providethe first analog signal to a camera control unit (CCU), convert thesecond image to a second analog signal, the second analog signal havinga second resolution different from the first resolution, and provide thesecond analog signal to the CCU. The system may further include atransmission medium to carry the analog signals from the camera head tothe CCU, such as a wired or wireless transmission medium. The CCU mayinclude a CCU body, an electronics module configured to receive andprocess analog signals including image or video data provided from thecamera head, an electronics module configured to extract the image orvideo data from the analog signals, and a CCU processing moduleconfigured to interpret the analog signal being received, or atranslated version of the analog signal, and format data containedtherein for display, storage, and/or transmission. The camera head mayfurther include a plurality of imaging elements and/or a lightingelement for generating markers on an area being viewed.

In another aspect, the disclosure relates to a camera head. The camerahead may include, for example, means for providing a first image from anelectronic image sensor at a first resolution to a camera control unit(CCU), means for sensing a condition associated with the image sensor,means for generating, responsive to the sensing, a second image from theimage sensor at a second resolution different from the first resolution,and means for providing the second image to the CCU. The camera head mayfurther include additional imaging means and/or means for projecting alight marker on an area being viewed.

In another aspect, the disclosure relates to a machine readable mediumincluding instructions for causing a processor or computer to receive afirst image at a first resolution from an electronic image sensor,convert the first image to a first analog signal, provide the firstanalog signal to a camera control unit (CCU), receive a sensor signalassociated with a sensed condition related to the image sensor, receivea second image at a second resolution different from the firstresolution from the image sensor, convert the second image to a secondanalog signal, and provide the second analog signal to the CCU. Theinstructions may further include instructions to project a light markeron an area being viewed. The instructions may further includeinstructions for registering images from the electronic image sensor anda second electronic image sensor based in part on the light marker.

In another aspect, the disclosure relates to a method of providing auser-controlled variable-resolution visual display in an inspectionsystem. The method may include, for example, capturing a first imagecovering a first field of view in an electronic image sensor of a camerahead. The method may further include converting the first image to afirst analog signal and providing the first analog signal to a cameracontrol unit (CCU) or other electronic computing system. The method mayfurther include receiving a control signal from the CCU and generating,at least partially in response to the control signal, ones of aplurality of tiled images corresponding to tiled subsets of the firstfield of view. The method may further include converting the pluralityof tiled images to a second analog signal. The method may furtherinclude providing the second analog signal to the CCU or otherelectronic computing system.

The method may further include, for example, receiving the first analogsignal, such as at the CCU or other electronic computing system,providing a first visual output corresponding to the first analog signalon a display device at a first resolution, receiving the second analogsignal, and combining the plurality of tiled images of the second analogsignal to provide a second visual output on the display device at asecond resolution higher than the first resolution. The control signalmay be generated based on a user input received at the CCU. The controlsignal may be generated at the CCU. The CCU may generate the controlsignal responsive to a processing analysis of motion based on previouslyreceived images or data from the camera head. The analysis may include adetermination of lack of motion in the previously received images.

In another aspect, the disclosure relates to a camera head. The camerahead may include, for example, a body and an electronic imaging element,including an image sensor and associated optics, disposed in the body.The image sensor may be configured to capture a first image covering afirst field of view and capture data for generating a second imageresponsive to a received control signal. The camera head may furtherinclude an electronics module. The electronics module may be configuredto receive a control signal from a CCU, convert the first image to afirst analog signal, provide the first analog signal to a camera controlunit (CCU), convert the second image to a second analog signal, wherethe second analog signal has a second resolution different from thefirst resolution, and provide the second analog signal to the CCU.

The camera head may, for example, include a second electronic imagingelement disposed in the body. The imaging elements may be oriented suchthat a field of view of the first imaging element may overlap with afield of view of the second imaging element. The optical axis of thefirst imaging element and the second imaging element may be disposed ina non-parallel orientation. The optical axes of the first imagingelement and the second imaging element may be divergent.

The camera head may, for example, include a plurality of additionalimaging elements. The imaging element may be oriented along a camerahead centerline, and additional imaging elements may be oriented withoutward divergent optical axes relative to the imaging element. Theoptical axes of ones of the plurality of additional imaging elements maybe non-parallel. The optical axes of ones of the plurality of additionalimaging elements may be divergent.

The camera head may, for example, include a lighting element to projecta light marker on an area being viewed by the camera head. The lightingelement may be a laser or other light emitting device. A grating orother optical element may be used to generate the target.

In another aspect, the disclosure relates to a camera control unit(CCU). The CCU may include, for example, a body, an electronics moduleconfigured to receive and process analog signals including image orvideo data provided from a camera head, an electronics module configuredto extract the image or video data from the analog signals, an analysismodule configured to receive a plurality of images or video from acamera head and generate a control signal to be provided to the camerahead, and a processing module configured to interpret the analog signalbeing received, or a translated version of the analog signal, and formatdata contained therein for display, storage, and/or transmission.

The processing module may, for example, be further configured to receivedata in conjunction with the video signal and use the data to display,store, and/or produce a visual display associated with the images orvideo. The analysis module may be configured to compare ones of aplurality of images or video previously received from the camera headfor motion and generate, based at least in part on the analysis, thecontrol signal based on the determined motion or lack of motion.

In another aspect, the disclosure relates to a pipe inspection system.The pipe inspection system may, for example, include a camera controlunit (CCU) and a camera head. The camera head may include a body and anelectronic image sensor disposed in the body. The image sensor may beconfigured to capture a first image covering a first field of view andcapture data for generating a second image responsive to a receivedcontrol signal. The camera head may further include an electronicsmodule. The camera head electronics module may be configured to receivethe control signal from a CCU, convert the first image to a first analogsignal, provide the first analog signal to a camera control unit (CCU),convert the second image to a second analog signal, the second analogsignal having a second resolution different from the first resolution,and provide the second analog signal to the CCU. The system may furtherinclude a transmission medium to carry the analog signals from thecamera head to the CCU. The CCU may include a CCU body, a CCUelectronics module to receive and process analog signals, includingimage or video data provided from a camera head, and extract the imageor video data from the analog signals, a CCU motion analysis moduleconfigured to receive a plurality of images or video from a camera headand generate a control signal to be provided to the camera headassociated with a desired motion, rotation, and/or translation, and aprocessing module configured to interpret the analog signal beingreceived, or a translated version of the analog signal, and format datacontained therein for display, storage, and/or transmission on or fromthe CCU.

In another aspect, the disclosure relates to a processor readablemedium. The processor readable medium may, for example, includeinstructions for causing a process or computer to capture a first imagecovering a first field of view, such as in an electronic image sensor ofa camera head. The instructions may further include instructions toconvert the first image to a first analog signal, provide the firstanalog signal to a camera control unit (CCU), receive a control signalfrom the CCU, generate, at least partially in responsive to the controlsignal, ones of a plurality of tiled images corresponding to tiledsubsets of the first field of view, convert the plurality of tiledimages to a second analog signal, and provide the second analog signalto the CCU.

Various additional aspects and details of self-leveling systems andmethods that may be used in embodiments in conjunction with thedisclosure herein are described in co-assigned U.S. patent applicationSer. No. 10/858,628, entitled SELF-LEVELING CAMERA HEAD, filed Jun. 1,2004, as well as co-assigned U.S. patent application Ser. No.13/358,463, entitled SELF-LEVELING INSPECTION SYSTEMS AND METHODS, filedJan. 25, 2012. The entire disclosure of each of these applications isincorporated by reference herein in its entirety.

Various additional features, aspects, and embodiment and implementationdetails are further described below in conjunction with the appendedDrawings.

Example Embodiments

FIG. 1A illustrates one embodiment 100 of such an inspection system inaccordance with various aspects. Inspection system 100 may include acamera head module 110 configured to mechanically house components suchas an imaging element including an electronic image sensor 112 andassociated optics (not shown in FIG. 1), an orientation and/or positionsensing module 114, an image processing module 116, a line driver module118, and/or other modules or components, such as power supplies,insulation elements, heating elements, defogging elements, lenselements, and the like (not shown). The camera head may be coupled to apush-cable or other mechanism for deploying or retracting the camerahead from a pipe or other cavity (not shown). In some embodiments, suchas those described subsequently herein with respect to FIG. 19, thecamera head may further include a plurality of imaging elements and/oradditional lighting elements, such as LEDs, lasers, and the like (notshown in FIG. 1).

The image sensor 112 may be, for example, a high resolution image sensorsuch as an OV9810 9-Megapixel 1080 HD Video Image Sensor, manufacturedby the Omnivision® Company, and/or a MT9P006 5-Megapixel HD Video ImageSensor, manufactured by the Aptina® Company, and/or other image sensorsknown or developed in the art. In one embodiment, the image sensor mayhave an element array of n×m pixels, where n×m may be 3488×2616. In oneembodiment, the image sensor may have an element array of n×m pixels,where n×m may be 2592×1944. Various other pixel array configurations mayalso be used in different embodiments.

Orientation module 114 may include, for example, a sensing componentsuch as an accelerometer, such as a three-axis accelerometer like theLSM303DLH, or nine-axis device, such as MPU-9150, manufactured byInvenSense®, and/or other position and/or orientation elements known ordeveloped in the art, such as compass sensors, inertial navigationsensors, and the like. Image processing module 116 may comprise, forexample, a programmable logic device such as a Field Programmable GateArray (FPGA), such as the Actel Igloo AGL250V2-CSG1961, Spartan®-6 FPGA(45-150 size), one or more microcontrollers or microprocessors, digitalsignal processors (DSPs), ASICs, memory devices, and/or other electronicand processing components capable of being configured to receive, store,and process images such as described subsequently herein. Camera head110 may further include one or more output circuits configured toprovide image and/or video output, such as a line driver 118 as shown.Additional position or location modules may also be included in camerahead 110, such as GPS modules or other radio-based location modules,optical location or positioning modules, and the like.

A transmission medium 122 may be used to connect the camera head 110 toa camera control module (CCU) 124, or other electronic computing systemor device, to transfer output image or video signals or data. In anexemplary embodiment the transmission medium 122 may be a wiredconductor configured to carry the output signal. However, in otherembodiments the transmission medium may be a wireless medium, fiberoptic medium, or other signal transmission medium. Camera control module124 may include various components for controlling and monitoring cameraoperation and information, such as image or video resolution, lighting,zooming, panning, rotation, and/or related operations or functions.

A display module 126 may be coupled to and/or incorporated in controlmodule 124 to render images and/or video provided from camera head 110.Control module 124 may include additional components such as memory forstoring video and/or images and/or associated data or metadata, as wellas components for providing other output such as digital video and/orimage files, hard copy, or other data or information. The camera controlmodule may further include one or more wired or wireless communicationmodules (not shown) for sending and/or receiving data from other devicesor systems, such as cellular data communication modules, Wi-Fi or Wi-Maxmodules, ISM-band modules, Ethernet modules, USB or Firewire modules, orother communications modules, such as GPS.

In an exemplary embodiment, display module 126 is capable of renderingan orientation-adjusted output image or video corresponding to anon-adjusted or raw image or video data provided from the image sensor112, such as on an LCD or other display element. For example, a sourceimage, such as example source image 310 as shown in FIG. 3, may begenerated and provided from image sensor 112. The source image may berotated, at an angular orientation theta (Θ) (as shown) relative to areference orientation, which may correspond with a gravity-basedvertical orientation (e.g., vertical or up-down with respect to theearth's gravitation force, g). The orientation may be sensed byorientation module 114, such as by a three-axis accelerometer or otherorientation sensor, which may provide output data or information toimage processing module 116 where it may be used to generate orientationadjusted output.

It is noted that the various signal processing functions describedherein may, in some embodiments, be implemented in CCUs or otherelectronic computing systems. However, as processing functionalityimproves and/or for size or other constraints, similar processing mayalternately be implemented in camera heads, in whole or in part.Likewise, signal processing described herein with respect to cameraheads may, in some embodiments, be similarly implemented in a CCU orother system or device communicatively coupled to the camera head.

FIG. 1B is a block diagram illustrating additional details of anembodiment of example image processing and control modules in a camerahead 130 of a pipe inspection system. These processing modules may beimplemented using one or more processing element. Data collection module170 (as shown in FIG. 1B) may include an image sensor 172 (or inembodiments with an image sensor array, multiple image sensors). Inaddition, data collection module may include an orientation andacceleration sensing module 174, which may include analog or digitalsensors such as compass sensors, accelerometers, gravitational sensors,location or position sensors, and the like. Image data captured by andgenerated in the imager 172 may be provided to an image processingmodule 152. Data from sensor module 174 may be provided to a dataprocessing module 156, which then may provide output to a control module154 for control of the sensor module 174 (and/or other modules not shownin FIG. 1B).

Image processing module 152 may receive commands from the CCU 124,and/or processing module 152 may run autonomously in the Camera Head130. Processed image data may be buffered in the Frame Buffer 158 forinsertion in a video standard frame generator module 162 forstandardized transmission, or can bypass the Frame Buffer to betransmitted as a more raw form of image data by multiplexor 164. Imagercontrol may be autonomously determined at imager control multiplexormodule 142 in the camera head by imager control module 144, and/or maybe controlled by the CCU 124 via control input module 148, through theTransmission Media 122 and the Line Receiver 132.

Orientation and acceleration data from sensor module 174 may beprocessed and autonomously controlled in the camera head 130. Variousaspects of the orientation/acceleration data processing functionality146 may also be controlled by the CCU 124. Processedorientation/acceleration data from data processing module 156 may beembedded in the vertical or horizontal blanking interval of standardizedvideo (NTSC, PAL, ITU-R BT.656 are some examples of video standards), ormay can be opportunistically transmitted in a transmit data multiplexor168 of image data and orientation/acceleration data to the line driver118, or can be transmitted alone. The CCU 124 may send image informationto be displayed to the display module 126.

FIG. 1C is a block diagram illustrating details of multiple data setsfrom multiple image sensors which may be processed together in anembodiment of a single camera head. For example, data collection element180 may be made up of multiple instances of data collection modules 170(FIG. 1B), with one as a master and others as slaves. Data collectionelement 180 has a master-slave organization where data Collection 170slaves 1, 2, 3, . . . , N may send data to the data collection 170master to transmit to data processing element 190. Data processingelement 190 may be composed of multiple data processing 140 blocks (FIG.1B) in a master-slave orientation. The data processing 140 master maycontrol the data processing 140 slaves and the data processing 140slaves may transmit information to the data processing master forfurther processing before transmitting data to the line driver 118. TheCCU 124 may control any of the data collection slaves or master throughdata processing element 190. The CCU 124 may control some or all of thedata processing slaves or master through processes in data processingblocks 140.

FIG. 2 illustrates details of an embodiment of an image processingmodule such as image processing module 116 of FIG. 1A. Module 116 mayinclude one or more processing elements 202, which may comprisemicroprocessors, microcontrollers, programmable logic devices, such asfield programmable gate arrays (FPGAs), application specific integratedcircuits (ASICs), and/or other programmable devices as are known ordeveloped in the art, along with memory and peripheral components. Inthe case of a microprocessor or similar or equivalent device, storagemedia 204 may be incorporated within and/or coupled externally toprocessor element 202 so as to store and provide instructions to theprocessor element 202 to perform the various signal processing functionsas described subsequently herein. In some cases, the instructions may beincorporated within or integral to the processor element(s) 202, such asby being implemented in logic in a device such as an FPGA.

In operation, the image processing module 116 may receive a source imageor sequence of images from image sensor 112 of FIG. 1A (or, inembodiments with multiple image sensors, from two or more of the imagesensors), such as via data connection 205, and store the source image,such as example source image 310 of FIG. 3, to a memory element 206,which may be, for example, a single memory array or multiple memoryarrays. The data may be transferred directly to the memory element 206,such as through direct memory access (DMA) via connection 205 and/or maybe received by processor element 202 and then stored in memory element206.

Processor element 202 may then generate an output image or sequence ofimages, such as in the form of a sequence of images or a video signal,where the output image or video signal is orientation-adjusted relativeto the source images. Examples of embodiments of processing so as togenerate the orientation adjusted output image(s) are describedsubsequently herein.

Image processing module 116 may further include an output video signalconverter module 208, which may be configured to generate, either aloneor in conjunction with the processor element 202, theorientation-adjusted output image(s) or video signal. For example, inone implementation, converter 208 includes a digital-to-analog converter(DAC) configured to receive pixels corresponding to theorientation-adjusted output image(s) or video signal and convert them toan analog or digital video signal in a standard or proprietary videoformat. For example, in an exemplary embodiment, a digital-to-analogconverter (not shown) may encode binary digital data into an analogvideo signal, such as a composite, component, S-Video, or other analogvideo signal, which may be, for example, in the National TelevisionSystems (NTSC), Phase Alternate Line (PAL) format, and/or other image orvideo formats known or developed in the art. In some embodiments,digital images or digital video signals may alternately be provided asan output.

The output may then be coupled to a line driver, such as line driver 118as shown in FIGS. 1A-1C, which may then amplify and/or filter the videosignal. This may be done to improve the quality of transmission of theoutput video signal over the transmission medium 122 to the cameracontrol module 124 and/or display module 126 (FIG. 1A). In an exemplaryembodiment, transmission medium 122 is a conductor configured to carryvideo and/or data signals; however, in some embodiments it may be afiber optic cable or other connection, such as a wireless connection.

After receipt of the output video signal at the CCU 124 (as shown inFIGS. 1A-1C), an output display corresponding to the orientationadjusted image or video may be provided. For example, the output may beprovided by rendering the orientation adjusted video output signal ondisplay device 126, and/or the image or video may be stored on acomputer storage media, provided as hard copy, output or transmitted toanother device or system, or otherwise used for analysis of the pipe,conduit, or other void or cavity being inspected. Orientation adjustmentperformed in and provided from the camera head, rather than in the CCUor other devices, may advantageously improve overall pipe inspectionsystem performance, increase system flexibility, reduce CCU and displaycomplexity, size, and/or cost, and provide other advantages.

FIG. 3 illustrates details of image processing as may be implemented,for example, by system 100 and image processing module 116 of camerahead 110. Image 310 is an example source image, such as may be providedfrom image sensor 112. The source image may be rotated at an angle Θrelative to an orientation reference, such as a gravitational forcevector g (directed downward or normal to the earth's surface). Forexample, if the image sensor/camera head 110 were to be rotated at angleΘ while positioned above the earth's surface, a rotated image, such asimage 310 may result. While the rotation may be apparent in image 310,if such a rotation occurs while the inspection system is in a pipe,conduit or other void, the rotation may not be apparent. In this case,orientation information as may be provided from orientation module 114may be used to generate an orientation adjusted output image(s) or videosignal in the camera head.

Image 320 illustrates additional details of features associated withimage 310. In particular, in an exemplary embodiment, a circular area322 may be defined within the boundaries of source image 310. Thecircular area may be characterized by a circular definitional parameter,such as a radius, circumference or diameter. Within circular area 322, asub-image area 324 may be defined. Sub-image area 324 may be a square orrectangular area, in which case a diagonal (not shown) will correspondwith a line drawn along the diameter of the circular area 322, therebyallowing sub-image area 324 to fit within circular area 322 at anyrotation angle. Although the example circular area 322 is shown withinthe boundaries of the source image 310 area, in some embodiments thecircular area may optionally be larger than the image area shown, suchas to provide a slightly wider angle of view in certain dimensions or tootherwise provide additional image detail or features.

Consequently, a sub-image may be generated or extracted from the sourceimage in a square or rectangular configuration at any rotational angle.If theta is known, such as from orientation module 114, an exampleorientation adjusted sub-image 330 may be generated or extracted fromthe example source image 310.

For example, in one embodiment, the rectangular rotated sub-image area324 may have a diagonal resolution that is less than the horizontal andvertical resolution of the image sensor; thus, the rectangular rotatedsub-image area 324 may rotate within the source image to any anglewithin circular area 322 to give a display image area 330. For example,if a device used to capture image has an element sensor array of n×m,where n×m is 1000×1000, and a display device used to view the displayimage only needs 640×480 pixels, then a sub-image area 324 with adiagonal of 800 pixels may be rotated within the source image 320 to anyangle, and the display image 330 (corresponding to sub-image 324) may betransmitted without rescaling or cropping. Examples of methods forextracting the sub-image as an output from image processing module 116are further described below with respect to FIG. 5.

Although the examples of FIG. 3 illustrate a sub-image area entirelywithin a circle within the image sensor, which may be used to providethe widest angle of imaging within the capabilities of the image sensor,in some embodiments other pixel areas within the image sensor mayalternately be provided in an output image or video signal. For example,in some embodiments, a fraction of the image area, such as a half,third, quarter, etc., may be provided in the output video signal in anorientation adjusted fashion. These will generally be of a narrowerangle than that shown in FIG. 3, although in some cases wider aspectratios may be used for the output images or video signals.

FIG. 4 illustrates details of an embodiment of a process 400 forgenerating an orientation adjusted output image, sequence of images orvideo signal, such as example sub-image 330 shown in FIG. 3. At stage410, a source image or images may be generated, such as describedpreviously with respect to FIGS. 1A-1C. The source image(s) may be, forexample, an image represented by a series of pixel values provided froma high definition image sensor, such as the OV9810 described previously.At stage 420, an orientation signal with orientation and/or positionaldata may be generated, such as from an accelerometer or otherorientation or position sensing device. The orientation signal may be agravitation orientation signal to provide an indication of offset of theimaging sensor from an up-down orientation. In some embodiments, anoutput sequence of images may be generated based on pixel data receivedfrom multiple image sensors, such as when a viewport is moved around aviewing area covered by multiple sensors and through overlap areas ofmultiple sensors. Examples of this are further described subsequentlywith respect to FIG. 30 through FIG. 31H.

The source image(s) and orientation signal may be provided to an imageprocessing module, such as a stage 430, where a second or output imageor images may be generated consistent with the orientation signal.Examples of various methods of generating the output signal are furtherdescribed subsequently herein. At stage 440, the second or output imagemay be provided as an output signal, such as in the form of an image,series of images, or video signal. The video signal may be, for example,converted from a digital to an analog signal format, such as a compositevideo signal, component video signal, S-video signal, or other analogvideo signal. Data may be further embedded in the output video signal,such as in a blanking or synchronization interval in the analog videosignal. The data may relate to orientation information, additionalsensor information, audio data, position or motion data, position dataassociated with the second/output video frame sizing and/or orientation,or other data or information. In some implementations, the video outputmay be provided as an orientation-adjusted digital video signal.

FIG. 5 illustrates details of an exemplary embodiment of imageprocessing to generate a second or output image that is orientationadjusted relative to a first or source image. In one implementation, oneor a plurality of pixel positions, such as pixel 524 (shown asrepresentative of one of a plurality of pixels), may be determined to beused to scan a memory, such as a single memory set/memory array whichmay correspond with memory element 206 of FIG. 2

The basics of NTSC and PAL are similar in that a quadrature amplitudemodulated (QAM) subcarrier carrying chrominance (I/Q) information isadded to the luminance (Y) video signal to form a composite videobaseband signal. For example, chrominance may be encoded using two3.579545 MHz signals that are 90 degrees out of phase to each other,known as in-phase (I) and quadrature (Q). These two signals, I and Q,are amplitude modulated, and added together. The pixel positions for acomposite output exist as points in time, and an instantaneous positionof the continuously scanning beam. Luminance (Y) and chrominance (I/Q)may be encoded from a plurality of scan lines 526 to produce an outputimage (which may correspond with image 330 of FIG. 3).

In one implementation, a pixel coordinate map 528 may be used toillustrate the pixels from an image sensor, such as image sensor 112.The image sensor may include a Bayer filter to determine the color andintensity of virtual image pixel 524, which may be extracted from ordetermined by elements of the pixel coordinate map 528, such as byselecting values of one or more of the elements of coordinate map 528and/or by generating the virtual image pixel as a function of one ormore element values of the coordinate map. The Bayer filter is a patternof individual filters, each of which are denoted with one letterselected from “R”, “G”, and “B” to indicate a red, a green, and a bluelight transmission characteristic, respectively (see, for example, U.S.Pat. No. 3,971,065 of Bayer, the content of which is incorporated byreference herein in its entirety). The precise geometry of the virtualimage pixel 524 can be mapped onto a grid 530 of fully and partiallycovered RGB array of the image sensor, where the virtual pixel may berepresented by color and intensity values, with the color and intensityof the virtual image pixel 524 depending on its positioning on the grid530. A coordinate 532 may be indicated at the vertex of a 2×2 RGB pixelblock (and/or by a single pixel or other pixel block configurations)falling within or in proximity to one of the virtual image pixels 524.In one implementation, pixels may be selected for the virtual imagebased on a nearest neighbor or other vicinity mapping from thecoordinate map to the virtual image pixel. Alternately, or in addition,pixel blocks may be averaged to calculate a color and an intensityassociated with each of the virtual image pixels 524.

A centroid 534 may be determined and used to indicate a geometric centerof the virtual image pixels 524. A neighboring coordinate 536 may bedetermined based on a relative location to the vertex of the pixel blocknearest to the centroid 534. This neighboring coordinate 536 may be usedto serve as the color and the intensity input for the virtual imagepixel 524. In this case, processing some or all of the image pixelswithin the area of virtual pixel 524 may improve signal to noise ratioin the output image, which may subsequently improve the image and/orvideo quality at the display device or on a stored image or videosignal.

In one exemplary embodiment, it may be desirable to have the diagonal ofthe rotated image be as close to the limiting dimension of the imagesensor as is mathematically possible. This may be done to generate anoutput video signal with the widest field or angle of coverage. As shownin FIG. 3, the diagonal may be selected to correspond to the diameter ofthe circular area 322, however, in some cases, a larger diagonal may beselected so as to provide slight cropping at the corners of the rotatedimage. For example, a diagonal size may be selected to be slightlylarger than the diameter of circle 322. In some embodiments, thecentroid may be offset in one or both directions from the center of theimage sensor, such as to image details or features offset from thecenter.

Alternately, or in addition, the diagonal size may be further selectedbased on the orientation signal provided from the accelerometer or otherorientation sensor. In this case, if the image is only slightly rotated,the diagonal size may be selected to be larger as there will be lessimage cropping at the edges (compared to, for example, 90 degreerotation). As such, the size of the subset of pixels (e.g., number ofpixels in the subset) may be based at least in part on the orientationsignal, with more pixels used in the case of orientations close to thevertical. For example, if the image sensor is oriented at or close tovertical (e.g., the image from the image sensor is only slightly rotatedfrom a level orientation) the size of the subset of pixels used may belarger than if the image sensor is rotated at a greater angle, whichwill then allow for a greater angle of coverage on the output videosignal. Conversely, at the extreme, with a rectangular image sensor, thesmallest pixel subset size would be selected when the sensor is rotatedin the range of 90 degrees or 270 degrees, which would provide anarrower angle of coverage.

In some implementations, for maximum usable field of view, the diagonalof the rotated sub-image (in the pixel space of the high resolutionimage the virtual pixels of the transmitted image may be larger andfewer in number) may be approximately equal to the smallest of (m,n) ofthe high resolution image. In some implementations it may be desirableto slightly crop the corners of the rotated sub-image so as to maximizethe field of view of the image inside the pipe, conduit, or othercavity.

In some implementations, output image size may be determined inaccordance with a relationship such as └√(p²+q²)┘<(m,n). In anotherimplementation, output image size may be determined in accordance withthe relationship └√(p²+q²)┘<(m,n)/2 or └√(p²+q²)┘<(m,n)/5 (where “└” and“┘” indicate an “upper ceiling”). Other output sizes may be used invarious other embodiments.

Processing in RGB color space is generally inefficient and consumessubstantial memory resources. Thus, to reduce consumption of memoryresources, RGB color space may be converted directly to NTSC color spacefor transmission as a composite video output signal, such as isillustrated with respect to FIGS. 1A-1C and FIG. 2. For example, RGBcolor space may be converted directly to YCbCr color space for digitalimage processing; however, 4:1:1, 4:2:0, or 4:2:2 YCbCr data may beconverted to 4:4:4 YCbCr data before undergoing conversion to NTSC orPAL signal format (YIQ or YUV). The transformations between each colorspace as disclosed herein, may be carried out by methods known to one ofskill in the art and routine modifications thereof, and/or according toprocedures found in, for example, expired U.S. Pat. No. 7,015,962,entitled INTEGRATED COLOR INTERPOLATION AND COLOR SPACE CONVERSIONALGORITHM FROM 8-BIT BAYER PATTERN RGB COLOR SPACE TO 12-BIT YCRCB COLORSPACE, issued Mar. 21, 2006; and “Video Demystified,” by Jack, Keith,2005, pp. 15-34 and 394-471, each of which is incorporated by referenceherein.

In various embodiments, a color space transformation, such as YCbCr,YUV, or YIQ or chroma sub-sampling, such as 4:2:2, 4:2:0, or 4:1:1 maybe used to produce an appropriate video output. These transformationsand/or sub-sampling may be performed before or after the image (e.g.,plurality of pixels) is stored in a memory such as memory element 206.

In another aspect, the disclosure relates to a camera head andassociated display system device, such as a camera control unit (CCU) orother device configured to provide, display, and/or store variableand/or adjustable resolution images and videos during an inspectionoperation such as a buried pipe inspection. The various aspectsdescribed below may be implemented separately in some embodiments or maybe combined with the level-adjustment and related embodiments describedpreviously herein. In some embodiments, an output sequence of images maybe generated based on pixel data received from multiple image sensors,such as when a viewport is moved around a viewing area covered bymultiple sensors and through overlap areas of multiple sensors. Examplesof this are further described subsequently with respect to FIG. 30through FIG. 31H.

Turning to FIG. 6, an example high resolution image sensor 600 isillustrated. Image sensor 600 may be a CMOS or CCD sensor or other imagesensor type as known or developed in the art. Image sensor 600 includesmultiple pixels (typically on the order of 5-20 million pixels ormegapixels) configured in N rows by M columns M and N may be equal insome sensor embodiments or may be different numbers depending on thesensor type, nominal aspect, ratio, etc. The pixels of sensor array 600may correspond directly with output pixels or may be mapped, such asusing a Bayer pattern or other sensor pixel mapping to correspondingoutput pixels. For purposes of explanation, it is assumed that thepixels as shown in example array 600 correspond directly with the samenumber of output pixels, however, this relationship may be variedthrough use of interpolation, color grids on top of the sensor elements,or other sensor processing techniques known or developed in the art.

In operation, the sensor array pixels may be mapped to correspondingscan lines of an image or video frame. An example of this is shown inFIGS. 7 & 8, where FIG. 7 illustrates the sensor array 600 andcorresponding pixels, while FIG. 8 illustrates a frame or sub-frame 800of a scanned analog video frame corresponding with image data generatedfrom the sensor. Frame 800 includes a plurality, S, of scan lines 820,which may be scan lines of standard video formats such as NTSC, PAL,SECAM, and the like. The scan lines may be interlaced or progressive,depending on the video format used. In some embodiments, the scannedlines may be digital equivalents of the analog video scan lines.

A subset of the image sensor pixels as shown in FIG. 7 as P rows perscan line may be converted to generate ones of the scan lines 820. Forexample, when the image sensor resolution is substantially larger thanthe corresponding resolution of the scanned video, as may be the casewhen converting megapixel sensor data to standardized frame resolutionssuch as 640×480, multiple rows, P, may be combined to generate each scanline. Other conversion methods between digital pixel values andcorresponding analog lines of video may also be used in variousembodiments.

In various embodiments, the full field of the image sensor (or imagesensor array), or, in some implementations, a portion of the full fieldof the image sensor, may be sub-divided into one or more tiles. This maybe done in implementations using a single image sensor (such as shown inFIG. 9A) or, in some embodiments, implementations using multiple imagesensors in an image sensor array (such as shown in FIG. 9B). Each tilecan be viewed as a collection of pixels of the image sensor (or imagesensor array). In implementations using a multi-sensor array, referencemarkers, such as light markers in the form of dots or symbols, may beprojected on the area being observed to allow combination of data frommultiple sensors.

For example, as shown in FIG. 9A, the field of a single image sensor 600may be divided into four tiles, denoted as tiles 902, 904, 906, and 908(with example pixels shown as squares—the size and number of pixels inFIG. 9A are shown larger than they would typically be for clarity.Further, a real image sensor would typically have many more pixels thanthe number of squares shown). In other embodiments, different numbers oftiles may be used depending on the resolution of the image sensor and/orthe desired output resolution. For example, the image sensor field maybe divided into 9 tiles, 16 tiles, or other tiled configurations invarious embodiments. In some cases the tiling may be non-symmetric,e.g., the field may be divided into 2 tiles, 6 tiles, etc., depending onthe image sensor type and/or desired output resolution in either thehorizontal dimension, vertical dimension, or both, as well as thedesired aspect ratio. While square pixels are shown, in some embodimentsother pixel shapes may be used depending on the configuration of theimage sensor. For purposes of explanation, a four tile configurationwill be described subsequently, however, it is not intended that thetiling configurations be limited to four tiles—other configurations canalso be used in various embodiments.

In some embodiments, image sensors may be grouped into an array of twoor more sensors (also denoted herein as an image sensor array ormulti-sensor array). These arrays may be in various configurationsincluding pluralities of image sensors. For example, FIG. 9B illustratesdetails of a multi-sensor array 900B including 4 image sensors, shown asimage sensors 1 through 4, with the array having M×N pixels. In thisimage sensor array, the center of the array is located adjacent to pixel(1,1) of sensor 4 (where the pixel coordinates begin at the upper leftof each image sensor). The image sensor array may be subdivided intomultiple tiles, such as, for example, the nine tile configuration ofFIG. 9B, where the tiles are denoted as tiles 910 through 918.

Image and video capture and processing methods as described herein mayalso be implemented in a camera head using an image sensor array havinga plurality of image sensors. In addition, output images and video maybe generated based on pixels from multiple image sensors, which may bestitched together in a processing element of the camera head and/or acoupled CCU or other computing device or system. The stitching may beaided with references, such as unique characteristics or features of thearea being imaged and/or using projected light markers or otherreferences. In addition, panning, zooming, rotation, and/or otheroperations may be implemented across image sensors in an image sensorarray. This may be done either by combining pixels from multiple sensorsor dynamically switching, such as on a detected sensor area transitionor overlap area, from pixels on one sensor to pixels on other sensors.

For example, FIG. 9C illustrates details of a zoom operation and thecorresponding initial tile viewport 921 and final tile viewport 923 (theviewports may be viewed as the area of view from which output imagesand/or video are taken at a particular time). Two image sensors of imagesensor array 900C are shown in FIG. 9C (i.e., imager 919 and 922), andtiles may be defined as a subset of the image sensor (or image sensorarray) to form an output viewport (e.g., an area of the imager that willbe provided as an output, such as on a CCU display or other videodisplay device, stored in memory as images or video, and the like).

In a typical configuration, output data from the pixels of the imagesensors may be stored in a memory, such as in a mapped memory in theimager or in a separate processing element or other module of the camerahead and/or CCU. Output signals may be generated by extractingappropriate pixels from the memory and then processing them to providethe desired output signals. During image processing operations, such asbased on positioning, location, or orientation information provided bysensors on the camera head (e.g., leveling outputs, etc.) and/or basedon control signals provided from the CCU or other coupled controldevice, the viewport position and corresponding viewed area within thepipe or other area being inspected may be changed. With the imagingsensor array fixed in position (relative to the pipe or other objectunder inspection), digital articulation of the camera head may beimplemented by moving the viewport around (i.e., by changing the pixelsused to generate the output and/or interpolating between pixels ordiscarding pixels) the pixels of the imaging sensor array. Similararticulation operations may also be done while the camera head is beingmoved, such as while the camera head is being moved into or out of apipe under inspection.

For example, as shown in FIG. 9C, the viewport area may be moved fromthe initial area 921 to area 923 by using the pixels on the path shown,which may be incremented pixel by pixel or by other increments (e.g., bymultiple pixels for faster panning, etc.), to move from position 921 toposition 923. In this example, output pixels from multiple image sensors(e.g., sensor 919 and sensor 922) may be used to generate the resultantoutput image from viewport 923. The pixels of the multiple sensors maybe registered to determine boundaries using light markers (not shown inFIG. 9C), such as laser projected dots or other markers.

In addition to the X and Y translations around the image sensor arrayshown in FIG. 9C, zoom operations (i.e. in or out) as well as rotateoperations may be done on the output viewport. The output viewport maybe set at a standard format, such as a standard SD or HD resolution, or,in some embodiments, in a non-standard format. Movements of the viewportmay be done in response to camera head movement, such as movementssensed by orientation, position, and/or location sensors, and/or inresponse to control commands from a CCU or other coupled control system.

In configurations with a single image sensor, the translation, zoom,rotation, and other operations may be done within image sensorboundaries. Alternately, in configurations with multiple image sensorssuch as shown in FIG. 9C, the translation, zoom, rotation, and otheroperations may be done across image sensor boundaries. For example, thefinal output image viewport of tile 923 may be a combination of pixelsfrom image sensor 919 and image sensor 922. Alternately, when multipleimagers are used, the output may be selectively switched from pixelsfrom one imager to those of another imager (such as by extractingcorresponding data values from the pixel array memory) upon transitionfrom one image sensor to another image sensor. As noted previously, thesensors may be registered based on features of the area being inspected,and/or based on projected light markers such as dots, or otherregistration mechanisms. The data corresponding with the viewport may bestored in a frame buffer, such as frame buffer 158 and may be outputusing frame buffer 158 and a video standard frame generator 162, such asthrough a transmit data multiplexor 168, line driver 118, andtransmission medium 122 to CCU 124 (as shown in FIG. 1A through 1C).

In some embodiments, multiple images, either from a single image sensoror, in some implementations, multiple image sensors, may be combinedeither in the camera head or CCU to create a high-dynamic range (HDR)image. This may be done by combining images from overlapping viewedareas of the image sensor or from multiple exposures taken from a singleimage sensor. These HDR images may then be used to generate an HDR datastream, such as in the form of video based on the HDR processing.

HDR images are typically created by capturing multiple images of aparticular area at different exposures, and intelligently combining theexposures to retain detail in both the highlight and shadow areas. Thismay be done by adjusting the luminance and chrominance of the multipleimages to approximate how the image is seen by the human eye. Themultiple images, of limited dynamic range, may be additively combinedinto a single image that yields a high contrast, computationally createdwith detail in both the highlight and shadow areas, and tone mapping maybe applied. Example methods and apparatus for capturing and processingHDR images and video are described in, for example, U.S. Pat. No.8,159,579, entitled HIGH DYNAMIC RANGE VIDEO, issued Apr. 17, 2012; U.S.Pat. No. 8,072,503, entitled METHODS, APPARATUSES, SYSTEMS, AND COMPUTERPROGRAM PRODUCTS FOR REAL-TIME HIGH DYNAMIC RANGE IMAGING, issued Dec.6, 2011; and U.S. Pat. No. 7,680,359, entitled METHODS FOR MERGINGDIGITAL IMAGES TO OBTAIN A HIGH DYNAMIC RANGE DIGITAL IMAGE, issued Mar.16, 2010, as well as a number of other patents and patent publications.The content of each of the above-described patents is incorporated byreference herein.

FIG. 9D illustrates an embodiment of an image sensor array 900D andassociated processing that may be used to generate HDR images andcorresponding video streams (based on the images). In the example array900D, the four image sensors (imagers 0 through 3) are oriented andinclude optics such that they image an overlapping area of the scenebeing viewed (denoted as the Overlapping Field of View in FIG. 9D). Togenerate an HDR image, the scene may be simultaneously imaged by each ofthe sensors, with the exposure of each adjusted to different values tocapture different areas of light and dark in the scene being viewed. Forexample, as shown, imager 0 is set at a low gain value, image 3 is setat a high gain value, and imagers 1 and 2 are set at in-between values.The output data from each of the imagers may then be combined in aprocessing module 928, such as through the use of image processingdetailed in the above-described incorporated applications, to generate aseries of output HDR images and/or HDR video comprising a sequence ofthe images. Other methods may also be used, such as sequentiallysampling the scene by cycling through the imagers (assuming the areabeing viewed is not changing significantly) to generate the outputimage. It is noted that the multi-imager method may best be used wherethe area being viewed is relatively distant from the image sensors,since each will see a slightly different view of the area being imaged.Alternately, in embodiments with a single image sensor, multipleexposures may be captured sequentially, and then subsequently combinedto generate a single HDR image, or series of images that may be combinedin a video stream. In some embodiments, the imager gain or exposure canbe changed on a frame by frame basis by imager control 144 or imagercontrol 148 (as shown in FIG. 1B), to generate a series of differentexposures of an image only slightly delayed in time. For example, aseries of three exposures can be taken repeatedly at 90 frames persecond, stored in the frame buffer 158, and extracted so that theoptimal exposure of each region of the three frame segments arerepresented in the video stream at 30 frames per second, or the threeframes may be combined in a weighted average to generate a video streamat 30 frames per second.

In addition to HDR image processing, in some embodiments, images fromtwo or more image sensors of an image sensor array may be captured andprocessed to generate stereoscopic output data, such as in the form ofstereoscopic pairs that are viewable on traditional stereoscopic viewingdisplays as two separate images, and/or on newer stereoscopic viewingdisplays that are configured to simultaneously or switchedly provideseparate left and right images or video frames from a single display. Inthis operating mode, images may be captured from two different imagesensors and, since the image sensors are offset in position from eachother, the images can be combined to generate stereo pairs and/orstereoscopic video streams. In some embodiments, the image sensors maybe oriented such that their optical axis is parallel, however, in anexemplary embodiment the optical axes of the image sensors andassociated optics, which may be denoted as an imaging element, arenon-parallel. For example, FIGS. 21-23 illustrate an exemplaryembodiment wherein the image sensors and optics are oriented withdivergent optical axes. In this configuration, stereo pairs and/orstereo video streams may be generated in the camera head using aplurality of imaging elements having non-parallel optical axes.

In addition, captured stereo pairs and/or video can be used to generatea 3D model or reconstruction of the pipe under inspection, such as bycapturing sequences of pairs while moving through a pipe and combiningthese captured pairs to generate a model, such as a wireframe or othercomputer aided design (CAD) model representative of the interior of thepipe. Methods and apparatus for generating 3D models from stereoscopicimages are described in, for example, U.S. Pat. No. 8,699,787, entitledMETHOD AND SYSTEM FOR GENERATING A 3D MODEL FROM IMAGES, issued Apr. 15,2014; U.S. Pat. No. 8,031,909, entitled METHOD AND APPARATUS FORPRODUCING 3D MODEL OF AN UNDERGROUND ENVIRONMENT, issued Oct. 4, 2011;United States Patent Application Publication 20080112610, entitledSYSTEM AND METHOD FOR 3D MODEL GENERATION, published May 15, 2008; andU.S. Pat. No. 5,633,995, entitled CAMERA SYSTEM & METHODS FOR EXTRACTING3D MODEL OF VIEWED OBJECT, issued May 27, 1997, as well as various otherreferences in the art. The content of the above-described patents andpatent publications is incorporated by reference herein. Modeledinformation may include, for example, shapes of interior pipe (or otherobject) features, dimensions, textures, and the like. Acoustic or othersensing may also be combined with optical information to generate themodels.

In some embodiments, images may be captured from multiple sensors andeither processed directly in the camera head (as processing powerimproves) and/or transferred to a CCU. If NTSC video formatting is used,standard frame rates would allow transmission of one tile or viewportarea data in 1/30 of a second. A 5 MP image could be sent as tiles inapproximately half a second, and with four imagers all four images couldbe sent in about two seconds. If 64 tiles are sent at 30 frames persecond, based on indexing of tiles and laser dots or other markers inthe images, the tiles can be processed to generate a 3D model. In someimplementations, a low resolution context image may be initially sent,and then tiled images may subsequently be sent. Alternately, if tilesare sent, they may be binned together at the CCU and then tiled down,depending on available processing power in the camera head and CCU.Further, a standardized video format, such as NTSC, PAL, etc., can beused as a video pipe (e.g., image data may be sent by using the analogvideo channel to create a digital pipe within the video portions of thesignal, as opposed to schemes that use the blanking area of the videosignal for data transmission). Further, raw Bayer sensor data or otherdata or information may be sent via such a pipeline (e.g., sending rawimage sensor data through the NTSC or other standard analog videochannel). Details of this concept are further illustrated in the exampleembodiment illustrated in FIG. 9F.

In some implementations where features of the area being viewed areknown, such as within a pipe where the pipe is cylindrical and has acircular or oval cross-section, light markers, such as laser dots orother projected markers, can be used as a predictive model to providesize, centerline orientation and pointing direction, and/or otherinformation to contribute to the model and/or selectively transition theoutput from one image sensor of an image sensor array to another imagesensor.

Likewise, when camera heads with optical elements using wide anglelenses and diverging optical axes are used, images can be generated bystitching together pixels from multiple image sensors to generate anextreme wide angle field of view, such as a field of view of greaterthan 180 degrees. The stitching may be aided by light markers projectedonto the area being viewed, such as laser dots or other light markings,and/or by recognizing common features seen from multiple imagers andcombining the pixels to overlap in the areas of these features. Further,projected dots or markers may then be processed out of the resultantimages or video based on location and/or distinguishing characteristics,such as color (e.g., bright red laser dots), shape, size, etc.

Alternately, or in addition, strobed light markers may be used, such asin synchronization with the image sensor frame rate or other imagingparameters, so that only some frames are captured when the dots or othermarkers are projected. For example, laser dots or other markers may bestrobed such that they are imaged only in alternating frames or in otherspecific frames (e.g., every 4^(th) frame, 30^(th) frame, etc.). Thismay be used to allow capture of frames both with the markers (e.g., withred or other dots, grids, or other marker shapes) and without themarkers so as to facilitate presenting output and/or storing images orvideo that do not include the markers. This may further allow removal ofdots or other markers by combining images having markers in them withprevious or subsequent images without the markers during imageprocessing in the camera head, CCU, or other electronic computing deviceor system.

FIG. 9E illustrates details of another aspect, wherein multiple imagers929A, 930A, and 931A are disposed in an imaging sensor array in a camerahead 932. Each of the image sensors may have a fairly wide angle fieldof view, but less than 180 degrees. When combined, the image sensorarray may have a field of view that is greater than 180 degrees. Inoperation, a processing element 934 in camera head 932 may intelligentlycombine data sets 929B-931B (generated from corresponding image sensors929A-931A) to corresponding mapped memory 929C-931C. The memory mapcontents may then be selectively extracted, such as by scanning with apointer that specifies which windowed viewport or tile (e.g., as VGA,SD, HD, non-standard, etc.) is a video frame for output at a given time.The window may be scanned around the image data, and a camera headprocessing element 934 or CCU processing element 936 may select when toswitch from image data from one image sensor to the next based on whatlocation provides the smoothest transition. This may be based on a knownor determined overlap area of view of the imaging sensors of the sensorarray. In an exemplary embodiment, there may be some or even substantialoverlap between sensor imaging areas. Alternately, in some embodimentsthe sensors may be oriented to provide approximately unique coverageareas that may form adjacent boundaries. A predefined distortioncorrection map of the lens/optics coupled to each image sensor of thesensor array may be stored in memory and used to correct distortion ofthe images generated by the sensors (e.g., to correct fisheye,pincushion, chromatic distortion, or other wide-angle lens distortionsor other optical distortions) to provide better cross-sensortransitions, stitched combinations, or other image processing.

Processing methods in the camera head processing element 934 or CCU 936may decide which imager's data to send for viewing that particular areain order to provide a desired transition, such as based on known ordetermined overlay areas, or other inputs, such as orientation,location, or positioning sensor data. Viewing of tiles at viewing angleswith no overlap may not require a decision since only one imager mayprovide information from that viewing area.

The selected image region may then be inserted into a standardized framegenerator 935 that can provide a video DAC 937 with digital video foranalog conversion and transmission 938. The CCU 939 can then display 942the tile as video, or send it to memory 940 for construction into ahigher resolution image/video 900B for display.

In another aspect, standardized analog video may be used to transmitdata. For example, FIG. 9F is a block diagram illustrating details ofone embodiment of standardized analog video being used as a generalpurpose digital transmission mechanism. Information collected by sensorsin the camera head 943, and/or other data or information, may be sent toa processing module that adds redundancy and error detection/correctiondata, such as in error encoding module 944. The digital data may then beinserted into the active and non-active video portions of a standardizedvideo frame generator in module 945. The video frames containing thegeneral purpose digital data may be converted to an analog video signalby a video DAC 946, and sent over a transmission media 947 to a videoADC 948. The resulting digital video may be received at a CCU 949, anderrors may be detected and corrected by utilizing theoverhead/redundancy information inserted in the camera head, in errorrecovery module 950, data processing module 951, and memory 952. Theresultant data may then be stored in the CCU and/or provided toperipheral devices 953. Using this configuration, a standardized ITU-RBT.656 and/or NTSC/PAL analog video signal may be used as a generalpurpose digital information pipe, such as in inspection systems betweena camera head and CCU.

FIG. 9G illustrates details of an embodiment of a process 900G forcontrolling panning or zooming in an image sensor array including aplurality of imaging elements (each of which includes an image sensorand associated optics). At stage 954, the camera head may enter apan/zoom mode, such as shown in FIG. 9E. This may be initiatedautomatically, such as based on an operating condition of acorresponding inspection system, and/or based on a user control input.If a user input is provided, the CCU may send a control signal to thecamera head (e.g., camera head 932) to digitally articulate the outputviewport and corresponding displayed area of the area being inspected.At decision stage 955, a decision may be made as to whether to controlthe imaged window or viewport from the CCU (YES) or from the camera headitself (NO), such as based on sensor or other information, such asorientation, position, location, and/or other sensor data.

Assuming the camera head is controlled by the CCU, at stage 956, areference position, such as the origin coordinates or other positioninformation of the starting tile may be sent. This may include anoverall coordinate reference and/or a specific image sensor references(of the plurality of image sensors in the image sensor array). Imagesensor data may then be sent for one or more frames. At stage 958, adetermination may be made as to whether the tile/viewport is approachinga boundary. This may be determined by reference optical marks, sensordata, image array reference data (e.g., sensor movement and directionand location within an overlap zone or boundary area) and/or other dataor information. If no image boundary is sensed, processing may return tostage 956 and either the previous tile/viewport reference used togenerate a subsequent image or a new controlled reference position maybe received from the CCU to allow transition to that new referenceposition. In some embodiments, transitions may be effected in a slow,continuous manner, such as by moving sequentially through intermediatepixels between the initial and final viewport positions. Alternately, insome embodiments, the output viewport may be switched rapidly to a newlyreceived viewport position (from the CCU).

At stage 960, a determination may be made as to what area of an adjacentimage sensor should be transitioned to (i.e., what area of the adjacentimage sensor best aligns with the current or most recent viewport area).The output viewport may then be transferred from pixels of the firstimage sensor to the closest matching pixels of the adjacent imagesensor. This may be done as a hard switch between image sensors and/ormay be done as a merge or combined pixel transition, where pixels ofboth image sensors are used to generate image and video data during atransition period. At stage 962, a decision may be made as to whether tocontinue in pan/zoom mode or enter a new mode. If pan/zoom mode is to becontinued, processing may return to stage 960 and/or to stage 956 tocontinue sending image data.

If camera head control is determined at stage 955 (i.e., NO state),processing may continue to stage 957, where the camera head may decidethe origin or reference of the tile to be transmitted. This may includean overall coordinate reference and/or a specific image sensorreferences (of the plurality of image sensors in the image sensorarray). Image sensor data may then be sent for one or more frames. Atstage 959, a determination may be made as to whether the tile/viewportis approaching a boundary. This may be determined by reference opticalmarks, sensor data, image array reference data (e.g., sensor movementand direction and location within an overlap zone or boundary area)and/or other data or information. If no image boundary is sensed,processing may return to stage 957 and either the previous tile/viewportreference used to generate a subsequent image or a new controlledreference position may be received from the CCU to allow transition tothat new reference position. In some embodiments, transitions may beeffected in a slow, continuous matter, such as by moving sequentiallythrough intermediate pixels between the initial and final viewportpositions. Alternately, in some embodiments, the output viewport may beswitched rapidly to a newly received viewport position (from the CCU).

At stage 961, a determination may be made as to what area of an adjacentimage sensor should be transitioned to (i.e., what area of the adjacentimage sensor best aligns with the current or most recent viewport area).The output viewport may then be transferred from pixels of the firstimage sensor to the closest matching pixels of the adjacent imagesensor. This may be done as a hard switch between image sensors and/ormay be done as a merge or combined pixel transition, where pixels ofboth image sensors are used to generate image and video data during atransition period. At stage 963, a decision may be made as to whether tocontinue in pan/zoom mode or enter a new mode. If pan/zoom mode is to becontinued, processing may return to stage 961 and/or to stage 957 tocontinue sending image data.

Turning to FIG. 10, a generalized process embodiment 1000 for processingtiled images in a camera head of a pipe inspection system isillustrated. Process 1000 may begin at stage 1010 where the camera headand associated inspection system components, such as a push-cable, etc.,are deployed into a pipe or other cavity being inspected. This may bedone by a powered cable reel system and/or by an operator manuallyfeeding the camera system into the pipe or other cavity.

At stage 1020, during a non-tiled or standard operating mode, data fromthe camera head image sensor may be generated and converted to an analogvideo signal in a standard mode, such as, for example, NTSC or PAL mode,and sent to a display system device such as a camera control unit via awired or wireless connection. Although analog video is typically used inexisting systems and is likely to remain in use for some time, in someembodiments the video signal may be in a standard mode, such as, forexample, NTSC or PAL mode, and sent to a display system device such as acamera control unit via a wired or wireless connection. Although analogvideo is typically used in existing systems and is likely to remain inuse for some time, in some embodiments the video output signal from thecamera head may be a digital video signal or other digital datasignaling.

At stage 1030, a sensing element, such as an accelerometer or othermotion, position, optical, audio, or other sensor output, may generate acondition signal associated with the camera head and/or a condition orstate within the pipe or other cavity. In an exemplary embodiment, thecondition is stoppage of movement of the camera head within the pipe,such as when an operator stops pushing or otherwise deploying the camerahead into the pipe. In other cases, the condition may be a speed state,such as a speed of movement of the camera head within the pipe droppingbelow a predefined threshold. In other cases, the condition may be auser input provided from the CCU or other display system, such as anoperator input provided to the camera head to change from standardoperating mode to an enhanced video mode (referring to 1032, 1034 or1036), such as the tiled, pan and zoom, or zoomed video modes describedsubsequently herein. The condition may also be sensed by, for example,sensing a vibrational output of an accelerometer or other inertialnavigation sensor (not shown) disposed in the camera head. In operation,when a camera head is deployed at the end of a push cable, vibrationssensed by the accelerometer during deployment of the cable typicallydecrease or stop, thereby providing an indication of motion stoppage.Motion sensors and associated processing elements may be configured andcalibrated based on particular motions or related signals, such asvibration signals, audio signals, etc. Predefined thresholds for thesesensor signals may be used to determine motions. Gyroscopic or compasssensors can similarly be used. Other sensing mechanisms may includedistance measurement elements, such as a cable counter in a cable reelon which the push cable may be stored, or other motion or positionmeasurement sensors or devices.

Assuming the condition occurs or an operator input is provided, at stage1040 the camera head and/or CCU may switch to an enhanced video mode. Inthe camera head, tiled image and/or zoomed video and/or pan and zoomvideo may be generated, such as described subsequently herein, and thenconverted to an analog signal and provided to the CCU or other displaysystem device. Images and/or enhanced video may be sent in this way atstage 1050 until the condition changes (e.g., the camera head startsmoving again or other conditions change) or a subsequent operator inputis provided at stage 1060. If no change occurs at stage 1060, enhancedvideo mode (labeled A, B, and C) operation may continue, such as, forexample, providing ongoing tiled mode analog video output from thecamera head. Alternately, if the condition changes at stage 1060,operation may return to standard mode operation at stage 1020.

In an exemplary embodiment, frame stacking and high dynamic rangetechniques and methods may be used when the camera head from stage 1010is stopped. For example, the exposure may be varied frame to frame.

FIG. 11 illustrates additional details of one embodiment 1100 of tiledmode operation for providing high resolution still mode images. Process1100 may begin at stage 1110, where a condition is checked for. Stage1110 may correspond with stage 1030 of FIG. 10. If the condition hasoccurred, such as, for example, the camera head has stopped moving asmay be determined at a control circuit 1111 based on velocityinformation 1113, which may be provided from an accelerometer or othermotion or position sensing device, high resolution still operating modemay be entered at the camera head and/or CCU. A mode override input1115, such as may be provided from an operator, may also be used tooverride automatic sensing and allow direct operator control of thecontrol circuit 1111.

During operation in the high resolution still mode, the image sensor maygenerate pixel data at stage 1130 which may then be processed at stage1140, such as in a processing element as described herein, to divide thesensed pixels into multiple tiles, such as the four tiles shown in FIG.9A or into fewer or typically more tiles. At stage 1150, each tile maybe converted to an analog (or, in some implementations, digital)standard definition video frame or sub-frame. The analog video signalmay then be sent to the camera control unit (CCU) 1121, where it maythen be stored, sent to other devices or system, and/or displayed on adisplay device 1123. In an exemplary embodiment, multiple tiles may beextracted from the analog video signal and recombined in the CCU togenerate a high-resolution image that may then be presented to a user onthe display device 1123.

FIG. 12 illustrates additional details of another embodiment 1200 oftiled mode operation for providing intermediate resolution still modeimages. Components of inspection systems as illustrated in FIG. 12 maycorrespond with analogous components as described in FIG. 11, such as,for example, control circuit 1111 corresponding with control circuit1211, CCU 1121 corresponding with CCU 1221, display 1123 correspondingwith display 1223, etc.

Process 1200 may begin at stage 1210, where a condition is checked for.Stage 1210 may correspond with stage 1030 of FIG. 10. If the conditionhas occurred, such as, for example, the camera head has stopped movingas may be determined at a control circuit 1211 based on velocityinformation 1213, which may be provided from an accelerometer or othermotion or position sensing device, intermediate resolution stilloperating mode may be entered at the camera head and/or CCU. A modeoverride input 1215, such as may be provided from an operator, may alsobe used to override automatic sensing and allow direct operator controlof the control circuit 1211.

During operation in the intermediate resolution still mode, the imagesensor may generate pixel data at stage 1230 which may then be brokeninto multiple tiles at stage 1240, The tiles may be then be reduced inresolution at stage 1245, by, for example, reducing the resolution ofthe tiles, reducing the number of tiles, etc. Stages 1240 and 1245 maybe implemented in a processing element as described herein. At stage1250, each tile may be converted to an analog (or, in someimplementations, digital) standard definition video frame or sub-frame.The analog video signal may then be sent to the camera control unit(CCU) 1221, where it may then be stored, sent to other devices orsystem, and/or displayed on a display device 1223. In an exemplaryembodiment, multiple tiles may be extracted from the analog video signaland recombined in the CCU to generate an intermediate resolution imagethat may then be presented to a user on the display device 1223.

FIG. 13 illustrates additional details of another embodiment 1300 oftiled mode operation for providing higher resolution video. In abandwidth constrained system operating at or near the bandwidthcapacity, the frame rate may be reduced corresponding to the increase invideo resolution. Components of inspection systems as illustrated inFIG. 13 may correspond with analogous components as described in FIG.11, such as, for example, control circuit 1111 corresponding withcontrol circuit 1311, CCU 1121 corresponding with CCU 1321, display 1123corresponding with display 1323, etc.

Process 1300 may begin at stage 1310, where a condition is checked for,which may be similar to that described with respect to FIGS. 11 and 12.Stage 1310 may correspond with stage 1030 of FIG. 10. If the conditionhas occurred, such as, for example, the camera head has stopped movingas may be determined at a control circuit 1311 based on velocityinformation 1313, which may be provided from an accelerometer or othermotion or position sensing device, intermediate resolution stilloperating mode may be entered at the camera head and/or CCU. A modeoverride input 1315, such as may be provided from an operator, may alsobe used to override automatic sensing and allow direct operator controlof the control circuit 1311.

During operation in the high resolution video mode, the image sensor maygenerate pixel data at stage 1330 which may then be broken into multipletiles at stage 1340, thereby resulting in a sequence of tiles, which mayeach be sent as a frame of a standard analog video signal (or, in someembodiments, a digital signal). Stages 1340 and 1345 may be implementedin a processing element as described herein. Once all the tiles from asingle frame have been transmitted, such as for example, at stage 1345,then the camera may acquire another frame and repeat 1330, 1340, and1345, or may alternately change modes, resolution, tile sizes, and/ornumber of tiles, etc. at a stage 1355.

For example, in the sensor configuration as shown in FIG. 9A, four tileswould be sent for each subsequent frame of high resolution video. In theCCU 1321, the tiled information may then be stored, sent to otherdevices or system, and/or displayed on a display device 1323. In anexemplary embodiment, multiple tiles may be extracted from the analogvideo signal and recombined (e.g., at four tiles/video frame) in the CCUto generate a high resolution video signal that may then be presented toa user on the display device 1323. As noted previously, the video outputresolution will typically be inversely related to the frame rate due tothe transmission of multiple tiles per output video frame.

FIG. 14 illustrates additional details of an embodiment 1400 of tiledimage sensor data processing as may be done in a camera head of aninspection system. Image frame 1480 may correspond with a sensed orcaptured image of a sensor element, such as sensor 600 of FIG. 6. Asshown, the image may be subdivided into a plurality of tiles that eachcovers a subset of the image field of view. In some cases the aggregateimage field of view may be less than that of the sensor, however, thefield of view will typically be selected to cover substantially all thefield of view of the sensor and associated optics so as to provide thewidest angle of coverage available.

In the example of FIG. 14, four tiles, 1481, 1482, 1483, and 1484 areshown. However, as noted previously other numbers and/or positions oftiles may be used in other embodiments.

Process 1400 may be begin at stage 1410 in response to the sensing of acondition or user input, such as the motion-related conditions describedpreviously herein. This may be, for example, after stage 1030 of FIG.10. At stage 1420, a full image frame may be captured or, in some cases,a smaller field of view that may then be further tiled. At stage 1430, afirst tile of the plurality of tiles may be converted to an analog videoframe or sub-frame. For example, as shown in FIG. 14, tile 1481 may beconverted to a corresponding analog frame 1491, which may be sent as ananalog video signal from the camera head to a CCU. At optional stage1432, data may be added to the video signal, such as in the blanking orsync intervals, or at other places in the video signal.

In some embodiments, additional data or information may be added to theanalog video signal and/or may be sent as a separate analog or digitalsignal to the CCU. For example, data may be added to blanking intervalsof the video, sync intervals, or during other available datatransmission intervals of the analog signal. The data may be, forexample, information representing the position, size, shape of the tileor other information related to the tile. Other information, such asadditional sensor data, timing information, re-assembly information, orother information may also be included in the data.

At stage 1440 the analog video signal including the first tile may besent. At stage 1450, a test may be made regarding whether all of thetiles have been sent. If not, additional tiles (e.g., tiles 1482, 1483,and 1484) may also be converted and sent to the CCU in a similarfashion. If all the tiles have been sent, processing may then return tothe pre-condition state or the transmission of the next frame of highresolution video may be started.

FIG. 15 illustrates additional details of an embodiment 1500 ofselective tile data processing as may be done in a camera head of aninspection system. Process 1500 may be used in combination with process1400 as illustrated in FIG. 14 to provide both multiple tiled andposition-specific tiled image processing. Image frame 1515 maycorrespond with a sensed or captured image of a sensor element, such assensor 600 of FIG. 6. An initial output analog video frame or frames1525 may be generated from data received from the image sensor, withsubsequent position-specific tiled areas, such as areas 1557 of imageframe 1555 as shown, converted to an analog video signal or signals.

Process 1500 may begin at stage 1510 with capture of an image frame froman electronic image sensor in a camera head, such as sensor 600 of FIG.6. At stage 1520, a full frame image (or, in some implementations, asubset of the full frame image) may be converted to an analog videosignal frame or frames, such as scanned analog video frame 1525 asshown. Optionally, at stage 1522, data may be added to an analog videosignal (e.g., in blanking interval, sync, etc.) At stage 1530, theanalog frame or frames may then be sent from the camera head to a CCU orother system display device.

At stage 1540, in response to the sensing of a condition or user input,such as the motion-related conditions described previously herein orbased on a user input, such as a user selection of a zoomed-in areaand/or position of the image frame, a tiled area, such as area 1557, maybe processed. For example, at stage 1550, a full frame of image data maybe sensed and, at stage 1560, a tiled sub-area, such as area 1557, maybe further processed at stage 1570 to generate an analog signal (or, insome implementations as described elsewhere herein, a digital datasignal). Additional data or information may optionally be added at stage1580, such as described previously with respect to stage 1432 of process1400. At stage 1590, the analog video signal corresponding to theselected tile may then be sent to the CCU or other system displaydevice. In some embodiments, the tiled image generation may be repeated,such as for a predetermined time interval or based on a condition orfurther user input. For example, at stage 1595, the steps may repeatstarting from the beginning at stage 1510, or alternatively, repeat fromstage 1550.

FIG. 16 illustrates details of an embodiment of a process 1600 forproviding a variable-resolution visual display in an inspection system,such as a pipe inspection system. Process 1600 may begin at stage 1610where a first image may be captured in an electronic image sensor of acamera head, with the first image covering a first field of view. Atstage 1620, the first image may be converted to a first analog signaland provided to a camera control unit (CCU) at stage 1630. Process 1600may further include sensing a condition associated with the image sensorat stage 1640, such as a motion condition or a user provided input. Atstage 1650, responsive to the sensing or user input, ones of a pluralityof tiled images corresponding to tiled subsets of the first field ofview may be generated, such as in a processing element of the camerahead. At stage 1660, the plurality of tiled images may be converted to asecond analog signal. The second analog video signal may be provided tothe CCU or other display device at stage 1670.

The imaging sensor may be a high resolution sensor and the first imagemay be captured at a high resolution. In some embodiments the firstimage and/or plurality of tiled images may be provided as a digitalsignal rather than an analog video signal.

The process 1600 may further include, for example, receiving the firstanalog signal at a system display device, such as a camera control unit(CCU), and providing a first visual output corresponding to the firstanalog signal on a display device at a first resolution. The method mayfurther include receiving the second analog signal and combining, suchas in a processing element of the CCU or other system display device,the plurality of tiled images of the second analog signal to provide asecond visual output on the display device. The second visual output maybe provided at a second resolution that is higher than the firstresolution.

The process 1600 may further include, for example, generating a secondplurality of tiled images corresponding to tiled subsets of the firstfield of view, converting the second plurality of tiled images to athird analog signal, providing the third analog signal to the CCU, andcombining the second plurality of tiled images of the third analogsignal to provide a third visual output on the display device at a thirdresolution different than the first and the second resolutions.

The second and/or third visual outputs may, for example, be provided asa zoomed-in field of view relative to the first visual output. The firstimage and/or second image may be provided as frames of a video signal.The first image and the aggregate of the tiled subsets may correspond tothe same field of view. The aggregate of the tiled subsets maycorrespond to a narrower field of view than the first image.Alternately, the aggregate of the tiled subsets corresponds to a widerfield of view than the first image. The first image and the aggregate ofthe tiled subsets may approximately correspond to the field covered bythe full frame of the image sensor. A subset of tiles may be used tocreate zoomed and/or panned out images. The camera head may beconfigured to send a sub-set of the tile grid if the CCU does notrequire all tiles to generate the zoomed in display.

The CCU may be configured to communicate information to the camera headto coordinate configuration and information transfer, in either one orboth directions. For example, the CCU may send a configuration signal tothe camera head to send only the tiles needed at the CCU or switchingoperating modes, while other operating modes only require the camerahead to communicate with the CCU.

The plurality of tiled images may include, for example, four tiledimages. Each of the four tiled images may correspond to approximatelyone quarter of the first field of view. Alternately, the plurality oftiled images may include nine tiled images and each of the nine tiledimages corresponds to approximately one ninth of the first field ofview. In other embodiments, different numbers of tiles, such as 2 tiles,6 tiles, 12 tiles, or other numbers of tiles may be used.

The condition at stage 1640 may, for example, correspond to a motion ofthe image sensor. The motion may be a stop motion, such as the stoppageof movement of the camera head within the pipe or other cavity.Alternately, or in addition, the motion condition may be a speedthreshold, such as a predefined maximum or minimum speed of the camerahead within the pipe or other cavity. Alternately, or in addition, themotion may be a start motion, such as a start of movement of the camerahead within the pipe or other cavity. The motion may be determined basedon signals such as vibration signals, audio signals, image signals, orother signals related to motion or position. For example, motion may beascertained from image data/signals. Motion or vibrational signals maybe sensed by devices such as accelerometers, compass sensors, counters,image sensors, or other sensing elements in the camera head, push cable,or associated components such as cable reels. Alternately, or inaddition, the condition may be a rotation of the image sensor relativeto a reference orientation. The reference orientation may be, forexample, an up-down gravitational orientation.

The process 1600 may further include, for example, providing dataassociated with the plurality of tiled images. The data may relate to aposition and/or size of the ones of the plurality of tiled images.Alternately, or in addition, the data may relate to a transmission modein use in the pipe inspection system, such as a high resolution ormedium resolution still mode or a high resolution reduced frame ratevideo mode. Alternately, or in addition, the data may relate to one ormore sensing conditions associated with the images being transmitted,such as information generated by motion sensors, position sensors,pressure or temperature sensors, audio or optical sensors, imagesensors, other sensors. The data may be sent in every video frame or maybe sent in only certain frames. The data may be sent in anothercommunication channel from the video signals, such as a second wired orwireless transmission medium.

The first image and/or the ones of a plurality of tiled images and/orthe corresponding higher resolution images or video may be stored on adigital storage medium, such as a flash memory or drive, a disc,removable storage media, Blu-Ray, DVD media, and the like. Alternately,or in addition, the first image and/or the ones of a plurality of tiledimages may be provided as a hard copy output.

The subset of the plurality of tiled images may, for example, be used togenerate the zoomed-in field of view of the second visual output. Thedata may relate to electronic image sensor status information and/orcamera head information at the time of image capture.

In an exemplary embodiment, images may be stored in the camera head forlater retrieval.

FIG. 17 illustrates details of an embodiment of a process 1700 forproviding a visual display in an inspection system, such as a pipeinspection system. Process 1700 may begin at stage 1710 where a firstimage from an electronic image sensor at a first resolution may beprovided to a camera control unit (CCU). At stage 1720, a first imagemay be provided from the camera head, at a first resolution, to the CCUor other electronic computing system. At stage 1730, a conditionassociated with the image sensor may be sensed. At stage 1740,responsive to the sensing, a second image may be generated from dataprovided from the image sensor at a second resolution different from thefirst resolution. At stage 1750, the second image may be provided to theCCU.

The process 1700 may further include, for example, providing a firstvisual output corresponding to the first image on a display device, andproviding a second visual output corresponding to the second image onthe display device.

The first image and the second image may, for example, be provided asframes of a video signal. The second resolution may be lower than thefirst resolution. The first image may correspond with approximately thefield covered by the full frame of the image sensor, and the secondimage may correspond with a subset of the field covered by the fullframe. The subset may be a tile of the full frame. The tile may beapproximately one quarter of the full frame. Alternately, the tile maybe approximately one ninth of the full frame. The tile may beapproximately centered within the field of view of the image sensor.

The process 1700 may further include, for example, receiving a tileselection signal from the CCU. The tile position may be based at leastin part on the tile selection signal. The tile size may be based atleast in part on the tile selection signal. The tile selection signalmay be a zoom-in signal. The tile selection signal may be a zoom-outsignal. The tile selection signal may be a position translation signal.The tile selection signal may be a rotation signal. The tile selectionsignal may be a combination of one or more of the above-describedsignals. The process 1700 may further include receiving a tile selectionsignal at the camera head from the CCU. The tile size and/or tileposition may be based at least in part on the tile selection signal.

In an exemplary embodiment, the tile position may be set and/or shiftedto any position on a pixel by pixel basis (e.g., panning). For example,a multi-axis user interface device, such as the user interface devicedescribed in U.S. patent application Ser. No. 13/110,910, entitled USERINTERFACE DEVICES, APPARATUS, AND METHODS, filed May 18, 2011, may beused to control pan position.

The condition may correspond, for example, to a motion of the imagesensor. The motion may be a stop motion. The motion may be a speedthreshold. The motion may be a start motion. Motion or vibrationalsignals may be sensed by devices such as accelerometers, compasssensors, counters, or other sensing elements in the camera head, pushcable, or associated components such as cable reels. The condition maybe a rotation of the image sensor relative to a reference orientation.The reference orientation may be an up-down gravitational orientation.The condition may be related to one or more of the above-describedmotions or rotations.

The first image may correspond, for example, with approximately the fullframe of the image sensor, and the second image may correspond with asubset of the full frame. The subset of the full frame may be selectedbased at least in part on the motion of the image sensor. The process1700 may further include providing data associated with the second imageto the CCU. The second image may correspond to a tiled subset of thefull frame, and the data may correspond to a position and/or size of thetiled subset.

FIG. 18 illustrates details of an embodiment of a process 1800 forproviding a variable-resolution visual display in an inspection system,such as a pipe inspection system, based on an automatically generatedCCU control signal. Process 1800 may begin at stage 1810 where a firstimage may be captured in an electronic image sensor of a camera head,with the first image covering a first field of view. At stage 1820, thefirst image may be converted to a first analog signal and provided to acamera control unit (CCU) at stage 1830. Process 1800 may furtherinclude receiving a control signal from the CCU at stage 1840. At stage1850, at least partially in response to the sensing or user input, onesof a plurality of tiled images corresponding to tiled subsets of thefirst field of view may be generated, such as in a processing element ofthe camera head. At stage 1860, the plurality of tiled images may beconverted to a second analog signal. The second analog video signal maybe provided to the CCU or other display device at stage 1870.

The process 1800 may further include, for example, receiving the firstanalog signal at the CCU, providing a first visual output correspondingto the first analog signal on a display device at a first resolution,receiving the second analog signal at the CCU, and combining theplurality of tiled images of the second analog signal to provide asecond visual output on the display device at a second resolution higherthan the first resolution.

The process 1800 may further include, for example, generating thecontrol signal at the camera control unit. The control signal may begenerated based on user input received at the CCU. Alternately, or inaddition, the control signal may be generated at the CCU using anelectronic analysis of motion based on previously received images orvideo frames from the camera head. The analysis may include adetermination of lack of motion in the previously received images. Thelack of motion may be determined based on changes in one or morefeatures of the previously received images or video frames.

The imaging sensor may be a high resolution sensor and the first imagemay be captured at a high resolution. In some embodiments the firstimage and/or plurality of tiled images may be provided as a digitalsignal rather than an analog video signal.

The process 1800 may further include, for example, receiving the firstanalog signal at a system display device, such as a camera control unit(CCU), and providing a first visual output corresponding to the firstanalog signal on a display device at a first resolution. The method mayfurther include receiving the second analog signal and combining, suchas in a processing element of the CCU or other system display device,the plurality of tiled images of the second analog signal to provide asecond visual output on the display device. The second visual output maybe provided at a second resolution that is higher than the firstresolution.

The process 1800 may further include, for example, generating a secondplurality of tiled images corresponding to tiled subsets of the firstfield of view, converting the second plurality of tiled images to athird analog signal, providing the third analog signal to the CCU, andcombining the second plurality of tiled images of the third analogsignal to provide a third visual output on the display device at a thirdresolution different than the first and the second resolutions.

The second and/or third visual outputs may, for example, be provided asa zoomed-in field of view relative to the first visual output. The firstimage and/or second image may be provided as frames of a video signal.The first image and the aggregate of the tiled subsets may correspond tothe same field of view. The aggregate of the tiled subsets maycorrespond to a narrower field of view than the first image.Alternately, the aggregate of the tiled subsets may correspond to awider field of view than the first image. The first image and theaggregate of the tiled subsets may approximately correspond to the fieldcovered by the full frame of the image sensor. A subset of tiles may beused to create zoomed and/or panned out images. The camera head may beconfigured to send a sub-set of the tile grid if the CCU does notrequire all tiles to generate the zoomed in display.

The CCU may be configured to communicate information to the camera headto coordinate configuration and information transfer, in either one orboth directions. For example, the CCU may send a configuration signal tothe camera head to send only the tiles needed at the CCU or switchingoperating modes, while other operating modes only require the camerahead to communicate with the CCU.

The plurality of tiled images may include, for example, four tiledimages. Each of the four tiled images may correspond to approximatelyone quarter of the first field of view. Alternately, the plurality oftiled images may include nine tiled images and each of the nine tiledimages corresponds to approximately one ninth of the first field ofview. In other embodiments, different numbers of tiles, such as 2 tiles,6 tiles, 12 tiles, or other numbers of tiles may be used. In anexemplary embodiment, some tile sizes may use binned pixels for improvedsecurity.

The process 1800 may further include, for example, providing dataassociated with the plurality of tiled images. The data may relate to aposition and/or size of the ones of the plurality of tiled images.Alternately, or in addition, the data may relate to a transmission modein use in the pipe inspection system, such as a high resolution ormedium resolution still mode or a high resolution reduced frame ratevideo mode. Alternately, or in addition, the data may relate to one ormore sensing conditions associated with the images being transmitted,such as information generated by motion sensors, position sensors,pressure or temperature sensors, audio or optical sensors, or othersensors. The data may be sent in every video frame or may be sent inonly certain frames. The data may be sent in another communicationchannel from the video signals, such as a second wired or wirelesstransmission medium.

The first image and/or the ones of a plurality of tiled images and/orthe corresponding higher resolution images or video may be stored on adigital storage medium, such as a flash memory or drive, a disc,removable storage media, Blu-Ray, DVD media, and the like. Alternately,or in addition, the first image and/or the ones of a plurality of tiledimages may be provided as a hard copy output.

The subset of the plurality of tiled images may, for example, be used togenerate the zoomed-in field of view of the second visual output. Thedata may relate to electronic image sensor status information and/orcamera head information at the time of image capture.

In another aspect, a camera head may include an imager including aplurality of imaging elements, each of which include image sensors andassociated optics. Lighting elements, such as LEDs or lasers, may alsobe included in the camera head to provide lighting markers on areasbeing inspected (e.g., inside pipes, conduits, cavities, or other voids)that may be used to process images/video generated by the imagingelements to provide alignment between pixel outputs from multiplesensors and the like. Such a camera head may be used to implementadditional functionally in various of the process embodiments describedpreviously herein (e.g., as described with respect to the processes ofFIGS. 4, 9A-9G, and 10-18).

For example, the processes may further include capturing a subsequentimage in a second electronic imaging element of the camera head, andgenerating a third analog signal. The third analog signal may be basedat least in part on the subsequent image. The third analog signal may bebased entirely on the subsequent image. The third analog signal may begenerated based on a combination of the subsequent image captured by thesecond image sensor and one or more pixels of a previous image capturedby the image sensor. The subsequent image may be captured during orsubsequent to a transition of a viewport or tile across an overlap areabetween the first image sensor and the second image sensor.

The processes may further include generating an output image sequence orvideo stream from two or more imaging elements in the camera head. Theoutput image sequence or video stream may be switched from pixel data ofa first imaging element of the two or more imaging elements to pixeldata from a second imaging element of the plurality of imaging elements,such as during transition of a viewport through an overlap region of thefield of view or coverage area of the imaging elements. The process mayinclude projecting, from the camera head, a light marker on an areabeing inspected. The light marker may be a laser dot or plurality oflaser dots. The light marker may be a target graphic generated by agrating or other optical element. The target, such a laser dot orplurality of laser dots or other graphic, may be strobed insynchronization with a video frame rate associated with the imagingelements or a processing element of the camera head. The strobing mayprovide the target during only certain image captures or video frames.

The processes may further include registering images from a first imagesensor of a plurality of image sensors with one or more other imagesensors of the plurality of image sensors using the projected target.The registering images may be done with images generated duringtransition of the viewport through the overlap region. The processes mayfurther include capturing a subsequent plurality of images in the camerahead at different exposures and generating, based on the subsequentplurality of images, a high dynamic range (HDR) image. The processes mayfurther include generating a plurality of HDR images and generating avideo stream based on the plurality of HDR images. The processes mayfurther include projecting, from the camera head, a light marker ortarget on an area being inspected. The processes may further includegenerating the HDR image based in part on registration of the images atdifferent exposures using the projected light marker or target.

The processes may further include capturing a subsequent image in asecond electronic image sensor of the camera head and generating astereoscopic image pair or stereoscopic video stream based on thesubsequent image and an additional image captured by the image sensor.The processes may further include projecting, from the camera head, alight marker or target on an area being inspected. The processes mayfurther include generating the stereo pair based in part on registrationof the subsequent image and additional image using the projected lightmarker or target.

The processes may further include capturing a subsequent image in asecond electronic image sensor of the camera head and generating astitched composite image or video stream based on the subsequent imageand an additional image captured by the image sensor. The processes mayfurther include projecting, from the camera head, a light marker ortarget on an area being inspected. The processes may further includegenerating the stitched composite image based in part on registration ofthe subsequent image and additional image using the projected lightmarker or target.

Turning to FIG. 19, camera head embodiment 1900 illustrates an examplecamera head having multiple imaging elements, which may be used toprovide digital articulation functionality based on image componentsprovided from multiple imaging elements. Such a camera head may be usedin inspection systems to provide functionality including generation ofoutput image sequences or video streams based on images captured usingtwo or more imaging elements, such as or subsequent to a transition of aviewport or tile across an overlap area between the first image sensorand the second image sensor. The output image sequence or video streammay be based on switching off pixel data of a first imaging element ofthe two or more imaging elements to pixel data from a second imagingelement of the plurality of imaging elements, such as during transitionof a viewport through an overlap region of the field of view or coveragearea of the imaging elements.

A camera head such as shown in FIG. 19 may further be configured toproject a light marker or target on an area being inspected. The lightmarker may be a laser dot or plurality of laser dots. The light markermay be a target graphic generated by a grating or other optical element.The target, such a laser dot or plurality of laser dots or othergraphic, may be strobed in synchronization with a video frame rateassociated with the imaging elements or a processing element of thecamera head. The strobing may provide the target during only certainimage captures or video frames. Images from a first image sensor of aplurality of image sensors may be registered with one or more otherimage sensors of the plurality of image sensors using the projectedtarget in a camera head such as camera head 1900 of FIG. 19. Theregistering images may be done with images generated during transitionof the viewport through the overlap region.

Further, a multi-sensor camera head such as camera head embodiment 1900may be configured to capture a plurality of images in the camera head atdifferent exposures and generate, based on the subsequent plurality ofimages, a high dynamic range (HDR) image. The camera head may furthergenerate a plurality of HDR images and generate an output video streambased on the plurality of HDR images. A light marker or target may beprojected from the camera head on an area being inspected. The HDRimages may be based in part on registration of the images at differentexposures using the projected light marker or target.

Further, a multi-sensor camera head such as camera head embodiment 1900may be configured to capture images on a first image sensor of thecamera head and a second image sensor of the camera head and generate astereoscopic image pair or stereoscopic video stream based on the pairof images. A light marker or target may be projected from the camerahead on an area being inspected. The stereo pair or stereo video streammay be based in part on registration of the images using the projectedlight marker or target.

Further, a multi-sensor camera head such as camera head embodiment 1900may be configured to capture images from multiple image sensors of thecamera head and generating a stitched composite image or video streambased on the multiple images. A light marker or target may be projectedfrom the camera head on an area being inspected or viewed. The stitchedcomposite image or video may be generated based in part on registrationof the multiple images using the projected light marker or target.

Returning to FIG. 19, in an exemplary embodiment, camera head 1900 mayinclude a rear housing assembly 1910, which may be mated with a frontbezel assembly 1920 to form the housing. The front bezel assembly 1920may include a front bezel 1922, which may be fitted with one or morewindows or ports, such as one or more imager windows 1924, which may beflat sapphire windows or other transparent windows, one or more laserwindows 1926, which may likewise be sapphire windows or othertransparent windows, and one or more LED windows, such as LED window1928, which may be similarly configured. In operation, the imagerwindows form part of an optics assembly of the imaging elements todirect light to the imaging sensors for capture of images of the areabeing inspected. The laser windows and/or LED windows may be used togenerate light to be projected onto the area being inspected, such asilluminating lighting from the LEDs, and targets or markers from thelasers, which may be LED lasers or other lasers. For example, the lightmay be to illuminate the area being inspected in a flood or spotlightfashion (e.g., from the LED windows) as well as to generate targets ormarkers, such as light dots, crosses, or other reference symbols, whichmay be generated by lasers and optionally associated optics, such asgratings, etc. Multiple image sensors in camera head 1900 may be used toimplement the multi-sensor processing functions described previouslyherein. In some embodiments, a forward oriented imaging element orelements may be disposed in the camera head to capture images throughone or more forward-looking windows or ports, such as window 1930 ofFIG. 19.

FIG. 20 is an enlarged detail view of the camera head embodiment 1900 ofFIG. 19, taken from the front view.

FIG. 21 is a section view of the camera head embodiment 1900 of FIG. 20,taken from line 21-21 (through imager window 1924). It is noted that theembodiment illustrated in FIG. 21 does not include details of theforward-looking imaging element as described with respect to FIG. 19,however, such an imaging element may be disposed in the camera head by,for example, changing the size or position of the other imaging sensorsshown, and/or rearranging their topology, etc. In an exemplaryembodiment, camera head 1900 may include one or more image sensors(imager) 2102, such as four imaging sensors in an exemplary embodiment(or 5 with a forward-looking imaging sensor window as shown in FIG. 19)which may be mounted to an imager PCBA assembly 2110. Imager PCBAassembly 2110 may be supported an imager PCBA carrier 2114. A lensassembly, such as lens assembly 2130, may be fitted with a lens mount2104. One or more LEDs, such as LED 2128, may be disposed on an innerLED PCBA 2126. LED window O-ring 2118 may be used to seal out moisture,particulate matter, and the like. Front bezel assembly 1920 may includea PCB capture ring 2116 and an outer LED ring PCBA 2122.

One or more PCBs, such as an imager processing PCBA 2142, a processingPCBA 2144, an encoder PCBA 2146, and a power PCBA 2148, may be disposedin the rear housing assembly 1910 and may include processing elements toperform the image processing described previously herein. One or moreconductive wires, such as ribbon cable 2108, may be used to electricallyconnect one or more PCBs, such as imager PCBA 2112 and imager processingPCBA 2142. Lens assembly 2130 and image sensor 2102 form an instance ofan imaging element and, as shown in FIG. 21, multiple imaging elementsare oriented in the camera heads to have non-parallel (in this examplediverging) optical axes (i.e., the optical axes diverge from a straightahead central axis or centerline of the camera head by an angle α(alpha), as shown).

FIG. 22 is a section view of the camera head embodiment 1900 of FIG. 20,taken from line 22-22 (through laser window 1926), further illustratinga laser module 2202, which may be used to generate target light dots ormarkers for projection on areas being inspected to facilitateregistration of images from multiple image sensors and/or to provideother image processing functions as described herein.

FIG. 23 illustrates additional details of camera head embodiment 1900 inan exploded view. As shown in FIG. 23, spacers 2305 may be used tooffset the PCBs 2144, 2146 and 2148.

FIG. 24 illustrates an exploded view of front bezel assembly 1900 ofFIGS. 19-23.

FIG. 25 illustrates an exploded view of image sensor PCB assemblyembodiment 2110 as shown in FIGS. 21-23.

FIG. 26 illustrates how output light dots or other markers or targets,such as from lasers in a camera head 2600, which may correspond withcamera head 1900 as described previously herein, may be used to createmarkers on a pipe 2615 for detecting shape, orientation, or other imageprocessing reference information. Markers or targets may be one or morelaser dots, or other shapes that may be generated by a grating, such ascrosses, bars or lines, rectangles, circles, or other reference shapesor graphics.

FIG. 27 illustrates an example cylindrical pipe 2700 cross section inwhich a camera head of an inspection system may be deployed. In thisexample, lasers are disposed on or within the camera head in a leveledorientation or aimed with the axis of the camera head parallel andcentered on the pipe. In this example, the camera includes an imagesensor array with four image sensors having overlapping fields of view(FOVs). Projected laser dots 2708, along with accelerometer and thelocation of each of the dots in the four image sensors can be used toestimate the shape 2704 of the pipe 2700.

FIG. 28 illustrates an example inspection camera operation showing howcertain areas within the entire FOV of the image sensor array (compositeimage) will have overlapping areas of view 2802 between the imagesensors. Some areas within the entire FOV will only be visible to asub-set of the image sensors, whereas some areas will be within the FOVof multiple or all image sensors. Areas with overlap can be used forvarious processing functions described herein, such as transitioning theviewport during pan/zoom operations or rotational operations, HDRprocessing, stereoscopic processing, 3D model generation, and the like.

FIGS. 29-31H illustrate additional details of image processing in amulti image sensor array including a plurality of imaging elements(image sensors, along with corresponding optics). The exampleconfigurations shown may be used to generate output image sequences orvideo streams from multiple image sensors as described previouslyherein, such as during translation movements of viewports across fieldsof view of different sensors.

FIG. 29 illustrates one example image sensor 2900 having an array ofpixels 2905 (shown as square boxes within the sensor). In this examplesensor, there are 20 columns by 16 rows of pixels, forming a 320 pixelarray (denoted as pixels 2905-1 through 2905-320). Pixel arraycoordinates are shown, with the ordering starting at the lower left ofthe image sensor (i.e., coordinates 1,1 to 20, 16 in the upper right). Alimited pixel array size is shown in FIG. 29 for purposes ofillustration, however, in an actual image sensor, the number of pixelswill typically be substantially larger, such as, for example, sensorswith pixel arrays including five million or more pixels. In addition,the aspect ratio and/or shape of the pixels may vary in different imagesensors, and the pixel array configuration need not necessarily besquare or rectangular, but rather may have circular, oval, or othershapes.

As described previously herein, in an image sensor array with wide angleoptics, the viewed areas of each imaging element may generally overlap(in some embodiments, the sensor coverage areas may abut or onlyslightly overlap, or in some implementations be non-overlapping, such asto cover a wider area of view with less detail, but they will generallybe overlapping to facilitate smooth transitions and/or registration ofimages across sensors). An example of this is shown in the sensor arrayembodiment 3000 of FIG. 30, where four instances of image sensor 2900are shown (denoted as image sensors 3010, 3020, 3030, and 3040, whichmay, for example, correspond to the four image sensors illustrated inthe camera head of FIG. 23), along with their respective areas ofcoverage, which overlap in overlap regions 3070 and 3080.

It is noted that, while the example embodiment of FIG. 30 includes fourinstances of the image sensors 2900 (and associated optics), in otherembodiments other numbers and arrangements of image sensors andassociated optics may alternately be used. For example, in someembodiments, the camera head may include a series of two or moreconcentric rings of imaging elements (not shown in FIG. 30). In someembodiments a single centered, forward facing imaging element (not shownin FIG. 30) may be included for a total of five imaging elements. Insome embodiments, two forward facing imaging elements may be used toprovide further stereoscopic imaging capability (e.g., the two forwardfacing imaging elements may capture a reference forward-lookingstereoscopic pair, which may be used in signal processing in conjunctionwith the images from the other sensors). The forward facing imagingelements may be aligned on parallel optical axes oriented along thecamera head centerline, or on divergent optical axes diverging about thecamera head centerline. In some embodiments, forward facing imagingelements may be oriented along convergent, rather than divergent,optical axes. In addition the image sensors may be rotated in otherorientations relative to each other (compared to the orientations shownin FIG. 30), such as centered along a circle about the camera headforward axes, or grouped in pairs on opposite sides of the camera head.

An optical distortion map for the optics/lenses associated with eachimage sensor may be predetermined, and stored in a memory of the camerahead and/or CCU or other electronic computing system to be applied tocaptured images during image processing and mapping of the outputs ofthe multiple imagers for generating the output images or video, such asdescribed previously with respect to FIGS. 9B-9G. For example, adistortion map may be applied to the image outputs of each of the imagesensors as part of the image processing performed to generate a 3Dsurface model of the inside of a pipe or other area being inspected, orto generate tiled images or video. Application of a correction map basedon predetermined distortion characteristics may also be used to correctfor lens distortions (e.g., fisheye effects or other wide angle lensdistortions) and/or to aid in registration of pixels from adjacentsensors when stitched together to form output images or videos based onpixels from two or more sensors.

As described previously, various image processing functions may beimplemented using pixels in the overlap region, such as stereoscopicprocessing, 3D model generation, HDR processing, and the like. Inaddition, as described previously, a sensor array such as array 3000 canbe used to generate output images having an extremely wide angle fieldof view (e.g., 180 degrees or more in camera heads using a flat port,such as a flat sapphire or other transparent port) by combining outputpixels from multiple sensors into a stitched composite image or video.Further, the output viewport or tile provided from the sensor array canimplement pan and zoom operations across sensor boundaries. This isfurther illustrated with respect to FIGS. 31A-31H, which illustrates anexample panning or translational digital articulation operation.

FIG. 31A illustrates an initial position 3111 of output viewport 3100,which in this case is in the upper left corner of image sensor 3. Basedeither on an automatic control signal or a CCU or other operatorprovided control signal (such as described in the process of FIG. 9G)the viewport may be moved to a different position in the image sensorarray coverage area, such as position 3112 as shown in FIG. 31B. If theviewport is moved within the boundaries of a single image sensor, theoutput pixels for each image and/or corresponding video stream may beprovided from the memory associated with the single sensor. Conversely,when the viewport is within an overlap region, such as overlap region3070, a decision may be made as to whether to transition the output topixels from another image sensor. As described previously herein, thismay be done by either switching (hard transition) the output entirely tothe pixels of the second sensor, or by soft transitioning, such as bycombining pixels from each sensor or merging the pixels together to formthe output images within the transition zone.

FIG. 31C illustrates a subsequent viewport position 3113, where theoutput is either switched entirely to the pixels of image sensor 1 orcombined or merged while the viewport is still within the overlapregion. FIG. 31D illustrates additional movement of the viewport toposition 3114, in this case where the viewport is entirely within thecoverage area of sensor 1, and the output pixels are taken entirely fromimage sensor 1.

FIGS. 31E-31H illustrate additional example translations of the viewport3100 to positions 3115 through 3118, while outputs are transitionedthrough image sensor 2 and image sensor 4.

The term “level” or “leveled” may be used herein to describe a conditionin which an object is substantially parallel relative to a horizontalplane and perpendicular to a gravitational force. For example, a flatsurface on the earth normal to gravity may be a level surface.

The term “leveling” may be used in the sense of bringing an object to aposition substantially parallel to a reference plane, that is, to makean object level, typically with respect to the ground and agravitational reference.

The term “rotated” may be used herein to describe a condition in whichan object is substantially parallel relative to a horizontal plane andperpendicular to gravity.

The term “camera head” may be used interchangeably with “camera,” or“camera module.” The term “camera head” may be used herein to describe adevice or module comprising an imaging element with an image sensoroperably coupled to a lens, and may further include related orsupporting electrical, optical, and/or mechanical components, such as abody or other structure and related components. The term “camera head”or “camera” may further be used herein to describe a compositioncomprising an image sensor operably coupled to a lens, an orientationsensor, a programmable device, such as a microprocessor,Field-Programmable Gate Array (FPGA), or DSP, as well adigital-to-analog converter and a line driver.

The term “high resolution image sensor” may be used herein to describethe resolution of the image sensor relative to the resolution of theremote monitoring system and/or display device. The term “highresolution image sensor” may be used herein to describe a semiconductordevice that detects energy in the near infrared, infrared, visible,and/or ultraviolet spectrums to be used for the formation of a displayedimage based on the detected energy. The detected energy may be used toform a single static image or a series of images (such as from a videocamera) that may provide a moving image. Detection within the imagesensor may be deployed in a planar arrangement in a two-dimensionalorientation, where the detection devices (e.g. detection pixels) may bein rows and columns for digital video, or in horizontal lines slantedslightly downward for analog video. The term “high resolution imagesensor” may refer to a complementary metal oxide semiconductor (CMOS), acharge-coupled detector (CCD), and/or other suitable high resolutionimage sensors or other detection devices.

The high resolution image sensor may be, for example, a complementarymetal oxide semiconductor (CMOS). The complementary metal oxidesemiconductor (CMOS) may have an element array of n rows of pixels by mcolumns of pixels (n×m) where n×m is at least 1600×1200.

The complementary metal oxide semiconductor (CMOS) may further have anelement array configuration such as, but not in any way limited to:1600×1200, 2048×1536, 2240×1680, 2560×1920, 3032×2008, 3072×2304,3264×2448, 3488×2616, 4368×2912, 5616×3744, and/or 13280×9184 pixels. Inone exemplary embodiment the complementary metal oxide semiconductor(CMOS) has an element array of 3488×2616. The complementary metal oxidesemiconductor (CMOS) may be, for example, an OV9810 9-Megapixel 1080 HDVideo Image Sensor, manufactured by the Omnivision® Company.

The term “accelerometer” is used to refer to a device that is capable ofproviding data related to a physical orientation and/or a position of animage sensor with respect to gravitational forces (“gravity”). Othersuitable examples of orientation sensing devices that may be used are,for example, an inclinometer, a gyroscope, a magnetometer, a tiltsensor, and/or other orientation and/or position sensing devices knownor developed in the art. An “accelerometer” may refer, for example, to athree-axis accelerometer, a two-axis accelerometer, and/or a one-axisaccelerometer. The term “three-axis accelerometer” may be a singleorientation sensor capable of measuring three perpendicular axes oracceleration and is interchangeable with three separate accelerometersarranged on three perpendicular axes.

The term “angular orientation” refers to the angle to which the imagesensor is oriented with respect to gravity, g, the image recorded by theimage sensor with respect to gravity, g, and/or the image to bedisplayed on a remote monitoring system with respect to gravity, g.Display orientation is generally independent of the physical orientationof the image sensor as might be sensed by an orientation sensor.

The term “field programmable gate array” or (“FPGA”) may be used hereinto describe a semiconductor device containing an array of programmablelogic components, such as logic blocks, and programmable interconnectsthere between. Logic blocks can be programmed to perform the function ofbasic logic gates and/or relatively more complex combinationalfunctions. The FPGA logic blocks may also include volatile and/ornon-volatile memory elements. A hierarchy of programmable interconnectsallows logic blocks to be interconnected and programmed after the FPGAis manufactured to implement any logical function.

The term “single memory array” may refer to an array of memory locationsof one or more memory devices sufficiently large to hold a single n×mimage or frame of video.

The term “composite video signal” may refer to a format or formats ofanalog video in which luminance data (brightness), chrominance data(color), and synchronization data (horizontal sync, vertical sync, andcolor reference bursts) are embedded in a single line-level signal.Analog video is discrete in the vertical dimension (there are distinctlines), but continuous in the horizontal dimension (every point blendsinto the next with no boundaries), hence there are no pixels in thisformat. The term “composite video signal” may refer to an analogencoding system or formatting standards for broadcast televisionsystems. The term “composite video signal” may refer to a standardanalog video format, such as Electronic Industry Association (ETA),National Television System Committee (NTSC), Comittee ConsultatifInternational Radiotelecommunique (CCIR), Phase Alternate Line (PAL),and/or Sequential Color with Memory (SECAM).

The composite video signal may be, for example, an NTSC signal, which isused in most of North America and South America. The NTSC standard mostcommonly employed is an interlaced system where each frame is scannedfor two fields at 262.5 lines per field, and is combined to display aframe of video with 525 horizontal scan lines slanting slightly downwardfrom left to right. NTSC scans at 29.97 frames per second, with 59.94fields displayed per second.

The term “field” refers to a set of even lines and/or odd lines. Onefield contains all the odd lines of the image; and the other fieldcontains all the even lines of the image. The odd and even lines aredisplayed sequentially, thus interlacing a full frame. One full frame isproduced by two interlaced fields, and is displayed approximately every1/30 of a second. Fields can be interlaced or progressively scanneddepending on the video standard used.

In various embodiments, digital self-leveling pipe inspection camerasystems consistent with the present invention are subject to a widevariety of modifications not described above. For example, the digitalself-leveling pipe inspection camera system may be utilized inapplications not associated with pipe inspection, such as assembly linemonitoring, endoscopy, and/or other inspection or analysis applications.

The word “exemplary” is used herein to mean “serving as an example,instance, or illustration.” Any aspect and/or embodiment describedherein as “exemplary” is not necessarily to be construed as preferred oradvantageous over other aspects and/or embodiments.

In some configurations, the various systems and modules include meansfor performing various functions as described herein. In one aspect, theaforementioned means may be a processor or processors and associatedmemory in which embodiments reside, and which are configured to performthe functions recited by the aforementioned means. The aforementionedmeans may be, for example, processors, logic devices, memory, and/orother elements residing in a camera head, camera control module, displaymodule, and/or other modules or components as are described herein. Inanother aspect, the aforementioned means may be a module or apparatusconfigured to perform the functions recited by the aforementioned means.

In one or more exemplary embodiments, the functions, methods andprocesses described may be implemented in hardware, software, firmware,or any combination thereof. If implemented in software, the functionsmay be stored on or encoded as one or more instructions or code on acomputer-readable medium. Computer-readable media includes computerstorage media. Storage media may be any available media that can beaccessed by a computer. By way of example, and not limitation, suchcomputer-readable media can include RAM, ROM, EEPROM, CD-ROM or otheroptical disk storage, magnetic disk storage or other magnetic storagedevices, or any other medium that can be used to carry or store desiredprogram code in the form of instructions or data structures and that canbe accessed by a computer. Disk and disc, as used herein, includecompact disc (CD), laser disc, optical disc, digital versatile disc(DVD), floppy disk and blu-ray disc where disks usually reproduce datamagnetically, while discs reproduce data optically with lasers.Combinations of the above should also be included within the scope ofcomputer-readable media.

It is understood that the specific order or hierarchy of steps or stagesin the processes and methods disclosed are examples of exemplaryapproaches. Based upon design preferences, it is understood that thespecific order or hierarchy of steps in the processes may be rearrangedwhile remaining within the scope of the present disclosure. Theaccompanying method claims present elements of the various steps in asample order, and are not meant to be limited to the specific order orhierarchy presented unless explicitly noted.

Those of skill in the art would understand that information and signalsmay be represented using any of a variety of different technologies andtechniques. For example, data, instructions, commands, information,signals, bits, symbols, and chips that may be referenced throughout theabove description may be represented by voltages, currents,electromagnetic waves, magnetic fields or particles, optical fields orparticles, or any combination thereof.

Those of skill would further appreciate that the various illustrativelogical blocks, modules, circuits, and algorithm steps described inconnection with the embodiments disclosed herein may be implemented aselectronic hardware, computer software, or combinations of both. Toclearly illustrate this interchangeability of hardware and software,various illustrative components, blocks, modules, circuits, and stepshave been described above generally in terms of their functionality.Whether such functionality is implemented as hardware or softwaredepends upon the particular application and design constraints imposedon the overall system. Skilled artisans may implement the describedfunctionality in varying ways for each particular application, but suchimplementation decisions should not be interpreted as causing adeparture from the scope of the disclosure.

The various illustrative logical blocks, modules, and circuits describedin connection with the embodiments disclosed herein may be implementedor performed in a processing element with a general purpose processor,special purpose processor, digital signal processor (DSP), anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA) or other programmable logic device, discrete gate ortransistor logic, discrete hardware components, or any combinationthereof designed to perform the functions described herein. A generalpurpose processor may be a microprocessor, but in the alternative, theprocessor may be any conventional processor, controller,microcontroller, or state machine, which may be programmed to performthe specific functionality described herein, either directly or inconjunction with an external memory or memories. A processor may also beimplemented as a combination of computing devices, e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core, or any other suchconfiguration.

As used herein, computer program products comprising computer-readablemedia including all forms of computer-readable medium except, to theextent that such media is deemed to be non-statutory, transitorypropagating signals.

Various modifications to the aspects described herein will be readilyapparent to those skilled in the art, and the generic principles definedherein may be applied to other aspects without departing from the spiritor scope of the invention. Accordingly, the scope of the invention isnot intended to be limited to the aspects shown herein, but is to beaccorded the widest scope consistent with the specification anddrawings, wherein reference to an element in the singular is notintended to mean “one and only one” unless specifically so stated, butrather “one or more.” Unless specifically stated otherwise, the term“some” refers to one or more. A phrase referring to “at least one of” alist of items refers to any combination of those items, including singlemembers. As an example, “at least one of: a, b, or c” is intended tocover: a; b; c; a and b; a and c; b and c; and a, b and c.

The previous description of the disclosed aspects is provided to enableany person skilled in the art to make or use the present disclosure.Various modifications to these aspects will be readily apparent to thoseskilled in the art, and the generic principles defined herein may beapplied to other aspects without departing from the spirit or scope ofthe disclosure. Thus, the present invention is not intended to belimited to the aspects and details shown herein but is to be accordedthe widest scope consistent with the appended claims and theirequivalents.

We claim:
 1. A pipe inspection camera head, comprising: a body; a sensormodule for sensing a condition associated with the body and providing asensor output signal in response to the sensed condition; an electronicimaging element, including an image sensor and associated optics,disposed in the body, the image sensor configured to: capture a firstimage covering a first field of view, and capture, responsive to thesensor output signal, data for deriving ones of a plurality of tiledimages corresponding to tiled subsets of the first field of view; and anelectronics module configured to: convert the first image to a firstanalog signal; provide the first analog signal to a camera control unit(CCU); extract the plurality of tiled images; convert the plurality oftiled images to a second analog signal; and provide the second analogsignal to the CCU.
 2. The camera head of claim 1, further comprising asecond electronic imaging element disposed in the body, wherein a fieldof view of the imaging element overlaps with a field of view of thesecond imaging element.
 3. The camera head of claim 2, wherein theoptical axes of the imaging element and the second imaging element arenon-parallel.
 4. The camera head of claim 3, wherein the optical axes ofthe imaging element and the second imaging element are divergent.
 5. Thecamera head of claim 1, further comprising a plurality of additionalimaging elements.
 6. The camera head of claim 5, wherein the imagingelement is oriented along a camera head centerline, and whereinadditional imaging elements are oriented with outward divergent opticalaxes relative to the imaging element.
 7. The camera head of claim 5,wherein the optical axes of ones of the plurality of additional imagingelements are non-parallel.
 8. The camera head of claim 7, wherein theoptical axes of ones of the plurality of additional imaging element aredivergent.
 9. The camera head of claim 1, further including a lightingelement to project a light marker on an area being viewed by the camerahead.
 10. The camera head of claim 9, wherein the lighting element is alaser.
 11. A pipe inspection system, comprising: a camera control unit(CCU); a camera head including: a camera head body; a sensor moduleconfigured to sense a condition associated with the body; an electronicimage sensor disposed in the body, the image sensor configured to:capture a first image covering a first field of view; and capture,responsive to the sensing, data to be used to derive ones of a pluralityof tiled images corresponding to tiled subsets of the first field ofview; and an electronics module configured to: convert the first imageto a first analog signal; provide the first analog signal to a cameracontrol unit (CCU); extract the plurality of tiled images; convert theplurality of tiled images to a second analog signal; and provide thesecond analog signal to the CCU; a transmission media to carry theanalog signals from the camera head to the CCU; wherein the CCUincludes: a CCU body; an electronics module configured to receive andprocess analog signals including image or video data provided from thecamera head; an electronics module configured to extract the image orvideo data from the analog signals; and a CCU processing moduleconfigured to interpret the analog signal being received, or atranslated version of the analog signal, and format data containedtherein for display, storage, and/or transmission.